WO2017129138A1

WO2017129138A1 - Data protection method and apparatus in data warehouse

Info

Publication number: WO2017129138A1
Application number: PCT/CN2017/072699
Authority: WO
Inventors: 阳方
Original assignee: 中兴通讯股份有限公司
Priority date: 2016-01-26
Filing date: 2017-01-25
Publication date: 2017-08-03
Also published as: CN106997368A

Abstract

Disclosed are a data protection method and apparatus in a data warehouse. The method comprises: receiving a user request, the user request carrying user identity information and an operation request, the operation request comprising one or more of: a table-level operation request, a row-level operation request, a partition-level operation request and a bucket-level operation request; judging whether the user identity information and the operation request are legitimate; if both the user identity information and the operation request are legitimate, judging an operation type of the operation request; and if the operation request is an unlicensed operation, encrypting added data or existing data in a data warehouse.

Description

Data protection method and device in data warehouse

Technical field

The present disclosure relates to the field of data security, for example, to a data protection method and apparatus in a data warehouse.

Background technique

Hive is a data warehouse infrastructure. Hive is a mechanism for storing, querying, and analyzing data stored in the Hadoop Distributed File System (HDFS). It maps traditional structured data tables to data files on HDFS and provides simple Class structured query language (Structured Query Language, SQL) for querying. Among them, SQL is also called Hibernate Query Language (HQL), and Hive can convert SQL statements into programming model (MapReduce) tasks to run, thus achieving large-scale data processing. The resource entity used to carry data in the Hive includes a database, a table, a partition, and a bucket. The database can be regarded as a collection of multiple tables, so the user data can be considered to be stored in three entity objects: a table, a partition, and a bucket. A column is a field in a table and is part of a partition. A column can be distributed across multiple partitions. Therefore, all operations in Hive can be viewed as operations on tables, columns, partitions, or buckets.

As a large-scale data warehouse and offline analysis platform, Hive has been widely used, but the data security issues that follow have not received enough attention. Among them, Hive's data security issues are reflected in the following aspects:

All users in Hive are parallel. There is no system super administrator. The system cannot be managed globally and cannot restrict access by some illegal users.

Each user in the Hive can authorize each other, but can only authorize the operation of the table. If the column, partition, or bucket cannot be authorized, the user can only be allowed or prohibited from accessing the specific table. Data is controlled at the column level, partition level, and bucket level, such as allowing or disabling only users to access several columns or accessing data for several buckets;

The data files corresponding to the data in the Hive are stored on the HDFS. For the files stored on the HDFS, the storage format used by Hive is the text format (TEXT) or the Record Columnar File (RCFile). If a Hive User did not get other Hive users Authorization, obtain the permission to query the table, column, partition or bucket created by other Hive users, but the Hive user can still obtain the data information by directly obtaining the underlying HDFS file, which is equivalent to bypassing the upper layer of Hive. The control mechanism poses a serious threat to the data security of the data warehouse.

In response to the above problems, no complete and systematic solution has been found.

Summary of the invention

Embodiments of the present disclosure provide a data protection method and apparatus in a data warehouse, which can perform access control and column, partition, or bucket level authority control on a data warehouse user, and combine encryption of user data to enable The data in the data warehouse is protected.

Embodiments of the present disclosure provide a data protection method in a data warehouse, including:

Receiving a user request, where the user request carries user identity information and an operation request, where the operation request includes a table level operation request, a column level operation request, a partition level operation request, and a bucket level operation request. One or more;

Determining whether the user identity information and the operation request are legal;

If the identity information and the operation request are both legal, determining an operation type of the operation request, where the operation type includes an authorization operation and an unauthorized operation;

If the operation request is an unauthorized operation, the new data or the existing data in the data warehouse is encrypted.

Optionally, in the foregoing solution, the determining whether the user identity information and the operation request are legal, including:

Determining, according to the user identity information, whether the user corresponding to the user identity information exists in a pre-stored white list, and if yes, the user identity information is legal;

Determining whether the permission to execute the operation request is a preset operation authority of the user, and if so, the operation request is legal.

Optionally, in the foregoing solution, the unauthorized operation includes: creating a table operation, a data import operation, a modify table operation, an encryption operation on the existing data, and a data query operation.

Optionally, in the foregoing solution, encrypting the newly added data in the data warehouse, including:

Obtaining the create table operation, and creating according to the structural information of the table carried in the create table operation table;

Determining whether the first encryption configuration information is carried in the creation table operation, where the first encryption configuration information includes one or more of a name of a table to be encrypted, a name of a column, a name of a partition, and a name of a bucket. Encryption algorithm and decryption algorithm;

If the creation of the table operation carries the first encryption configuration information, storing the first encryption configuration information in the encryption information table;

Obtaining the data import operation, where the data import operation carries one of a name of a first target table, a name of a first target column, a name of a first target partition, and a name of a first target bucket that need to import data. Or a plurality, the first target table includes one or more tables in the created table, the first target column includes one or more columns of the created table, and the first target partition includes the created table One or more partitions, the first target bucket including one or more buckets of the created table;

Determining whether one or more of the first target table, the first target column, the first target partition, and the first target bucket store a corresponding encryption algorithm in the encryption information table;

If yes, obtaining an encryption algorithm corresponding to one or more of the first target table, the first target column, the first target partition, and the first target bucket;

Obtaining data to be imported into one or more of the first target table, the first target column, the first target partition, and the first target bucket, and using the obtained encryption algorithm to compare data to be imported Encrypting to obtain the first encrypted data;

And correspondingly writing the first encrypted data into one or more of the first target table, the first target column, the first target partition, and the first target bucket.

Optionally, in the foregoing solution, encrypting the existing data in the data warehouse, including:

Obtaining the modification table operation;

Determining whether the modification table operation carries the second encryption configuration information, where the second encryption configuration information includes a name of the second target table that needs to be encrypted, a name of the second target column, a name of the second target partition, and a second One or more of the names of the target buckets, an encryption algorithm, and a decryption algorithm;

And if the modification table operation carries the second encryption configuration information, using the encryption algorithm carried in the modification table operation, the second target table, the second target column, the second target partition, and the Encrypting data in one or more of the second target buckets;

The second encrypted configuration information is stored in the encrypted information table.

Acquiring the encryption operation on the existing data, where the encryption operation on the existing data carries the name of the table to be processed and the third encryption configuration information, where the third encryption configuration information includes the third target table that needs to be encrypted. One or more of a name, a name of a third target column, a name of a third target partition, and a name of a third target bucket, an encryption algorithm, and a decryption algorithm, the third target table including in the to-be-processed table One or more tables, the third target column including one or more columns of the pending table, the third target partition including one or more partitions of the pending table, the third target bucket Include one or more buckets of the pending table;

Creating a temporary table having the same structure as the pending table;

Obtaining the data query operation, and querying one or more of the third target table, the third target column, the third target partition, and the third target bucket according to the data query operation Data, obtain query data;

Correspondingly writing the query data into the temporary table;

Encrypting the query data written in the temporary table by using the encryption algorithm carried in the encryption operation of the existing data to obtain the second encrypted data;

Writing the second encrypted data into the to-be-processed table in an overlay manner;

Storing the third encrypted configuration information in the encrypted information table;

Delete the temporary table.

Optionally, in the above solution, the acquiring the data query operation, and querying the third target table, the third target column, the third target partition, and the third according to the data query operation The data in one or more of the target buckets, the query data is obtained, including:

Acquiring the data query operation, where the data query operation carries the name of the third target table, the name of the third target column, the name of the third target partition, and the name of the third target bucket One or more

Reading one or more of the third target table, the third target column, the third target partition, and the third target bucket from a distributed file system according to the data query operation The data;

Determining the third target table, the third target column, the third target partition, and the third Whether there is encrypted data in one or more of the target buckets;

And if so, obtaining, from the encrypted information table, a decryption algorithm corresponding to one or more of a table, a column, a partition, and a bucket in which the encrypted data is located;

Using the decryption algorithm, the encrypted data is decrypted to obtain the query data.

Optionally, in the foregoing solution, after determining the operation type of the operation request, the method further includes:

If the operation request is an authorization operation, determining whether one or more of the table, column, partition, and bucket in which the user carried in the authorization operation authorizes other users to operate belongs to the user;

If yes, the authorization operation is performed, and if not, the authorization operation is cancelled.

Optionally, in the foregoing solution, the method further includes:

The data in the encrypted information table is encrypted by using a preset encryption algorithm.

The embodiment of the present disclosure further provides a data protection device in a data warehouse, including:

The receiving module is configured to receive a user request, where the user request carries user identity information and an operation request, where the operation request includes a table level operation request, a column level operation request, a partition level operation request, and a bucket level. One or more of the operational requests;

a first determining module, configured to determine whether the user identity information and the operation request are legal;

a second determining module, configured to determine, when the first determining module determines that the identity information and the operation request are both legal, the operation type of the operation request, where the operation type includes an authorized operation and an unauthorized Operation;

The encryption module is configured to encrypt the newly added data or the existing data of the data warehouse when the second determining module determines that the operation request is an unauthorized operation.

Optionally, in the above solution, the first determining module includes:

The first determining unit is configured to determine, according to the user identity information, whether the user corresponding to the user identity information exists in a pre-stored white list, and if yes, the user identity information is legal;

The second determining unit is configured to determine whether the permission to execute the operation request is a preset operation authority of the user, and if yes, the operation request is legal.

Optionally, in the foregoing solution, the encryption module includes:

a first obtaining unit, configured to acquire the create table operation, and create a table according to the structural information of the table carried in the create table operation;

The third determining unit is configured to determine whether the first encryption configuration information is carried in the creation table operation, where the first encryption configuration information includes a name of a table to be encrypted, a name of a column, a name of a partition, and a name of a bucket. One or more, an encryption algorithm and a decryption algorithm;

a first storage unit, configured to: when the third determining unit determines that the creation table operation carries the first encryption configuration information, storing the first encryption configuration information in the encryption information table;

a second obtaining unit, configured to acquire the data import operation, where the data import operation carries a name of a first target table that needs to import data, a name of the first target column, a name of the first target partition, and a first target One or more of the names of the buckets, the first target table comprising one or more tables in the created table, the first target column comprising one or more columns of the created table, the first The target partition includes one or more partitions of the created table, the first target bucket including one or more buckets of the created table;

a fourth determining unit, configured to determine whether one or more of the first target table, the first target column, the first target partition, and the first target bucket are in the encryption information table Storing a corresponding encryption algorithm;

a third acquiring unit, configured to: when the fourth determining unit determines one or more of the first target table, the first target column, the first target partition, and the first target bucket Acquiring one or more of the first target table, the first target column, the first target partition, and the first target bucket when a corresponding encryption algorithm is stored in the encryption information table Corresponding encryption algorithm;

a first encryption unit, configured to acquire data to be imported into one or more of the first target table, the first target column, the first target partition, and the first target bucket, and utilize the acquired The encryption algorithm encrypts the data to be imported to obtain the first encrypted data;

a first writing unit, configured to write the first encrypted data correspondingly to one of the first target table, the first target column, the first target partition, and the first target bucket Or a variety of.

Optionally, in the foregoing solution, the encryption module includes:

a fourth obtaining unit, configured to acquire the modified table operation;

The fifth determining unit is configured to determine whether the second encryption configuration information is carried in the operation of the modification table, where the second encryption configuration information includes a name of the second target table that needs to be encrypted, a name of the second target column, and a second One or more of the name of the target partition and the name of the second target bucket, an encryption algorithm, and a decryption algorithm;

a second encryption unit, configured to: when the second determination unit carries the second encryption configuration information in the operation of the modification table, using the encryption algorithm carried in the modification table operation on the second target table, Encrypting data in one or more of the second target column, the second target partition, and the second target bucket;

The second storage unit is configured to store the second encrypted configuration information in the encrypted information table.

Optionally, in the foregoing solution, the encryption module includes:

The fifth obtaining unit is configured to obtain the encryption operation on the existing data, where the encryption operation on the existing data carries the name of the to-be-processed table and the third encryption configuration information, where the third encryption configuration information includes One or more of an encrypted third target table name, a third target column name, a third target partition name, and a third target bucket name, an encryption algorithm, and a decryption algorithm, the third target table including One or more tables in the to-be-processed table, the third target column includes one or more columns of the to-be-processed table, and the third target partition includes one or more partitions of the to-be-processed table The third target bucket includes one or more buckets of the to-be-processed table;

Creating a unit, set to create a temporary table having the same structure as the pending table;

a query unit, configured to acquire the data query operation, and query one of the third target table, the third target column, the third target partition, and the third target bucket according to the data query operation Data in one or more kinds, obtaining query data;

a second writing unit, configured to write the query data into the temporary table correspondingly;

The third encryption unit is configured to encrypt the query data written in the temporary table by using the encryption algorithm carried in the encryption operation on the existing data to obtain the second encrypted data;

a third writing unit, configured to write the second encrypted data into the to-be-processed table in an overlay manner;

a third storage unit configured to store the third encrypted configuration information in the encrypted information table;

Delete the unit, set to delete the temporary table.

Optionally, in the foregoing solution, the query unit includes:

a first obtaining subunit, configured to acquire the data query operation, where the data query operation carries a name of the third target table, a name of the third target column, a name of the third target partition, and a location One or more of the names of the third target buckets;

Querying a subunit, configured to read from the distributed file system the third target table, the third target column, the third target partition, and the third target bucket according to the data query operation Data in one or more;

a determining subunit, configured to determine whether the encrypted data exists in one or more of the third target table, the third target column, the third target partition, and the third target bucket;

a second obtaining subunit, configured to determine one or more of the third target table, the third target column, the third target partition, and the third target bucket when the determining subunit determines And obtaining, in the encrypted information table, a decryption algorithm corresponding to one or more of a table, a column, a partition, and a bucket in which the encrypted data is located; and

The decryption subunit is configured to decrypt the encrypted data using the decryption algorithm to obtain the query data.

Optionally, in the above solution, the device further includes:

And an authorization module, configured to: when the second determining module determines that the operation request is an authorization operation, determine one or more of a table, a column, a partition, and a bucket that the user carried in the authorization operation authorizes other users to operate Whether it belongs to the user, if yes, the authorization operation is performed, and if not, the authorization operation is cancelled.

Optionally, in the above solution, the encryption module further includes:

The fourth encryption unit is configured to encrypt the data in the encrypted information table by using a preset encryption algorithm. The present disclosure also provides a non-transitory computer readable storage medium storing computer executable instructions arranged to perform the above method.

The present disclosure also provides an electronic device, including:

At least one processor;

a memory communicatively coupled to the at least one processor; wherein

The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to cause the at least one processor to perform any of the methods described above.

The data protection method in the data warehouse of the embodiment of the present disclosure can control the access of the data warehouse user by preventing the legality of the user identity information and the operation request carried in the received user request, and preventing the access of the illegal user; User data can be flexibly manipulated through table-level, column-level, partition-level, and bucket-level operational requests; and data in the data warehouse can be protected by encrypting user data. Therefore, the data protection method in the data warehouse of the embodiment of the present disclosure can protect data in the data warehouse.

DRAWINGS

1 is a flow chart showing a data protection method in a data warehouse of a first embodiment of the present disclosure;

2 is a block diagram showing the structure of a data protection device in a data warehouse according to a second embodiment of the present disclosure;

3 is a second structural block diagram of a data protection device in a data warehouse according to a second embodiment of the present disclosure;

4 is a block diagram showing the structure of a data protection device in a data warehouse according to a third embodiment of the present disclosure;

5 is a flowchart showing an application of a data protection device in a data warehouse in a third embodiment of the present disclosure;

6 is a first branch flow chart showing an application of a data protection device in a data warehouse of a third embodiment of the present disclosure;

Figure 7 is a flow chart showing a second branch of the application of the data protection device in the data warehouse of the third embodiment of the present disclosure;

8 is a third branch flow chart showing an application of a data protection device in a data warehouse of a third embodiment of the present disclosure;

Figure 9 is a flowchart showing a fourth branch of the application of the data protection device in the data warehouse of the third embodiment of the present disclosure;

FIG. 10 is a block diagram showing the hardware structure of an electronic device according to a fifth embodiment of the present disclosure.

detailed description

Exemplary embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. The technical features in the following embodiments and the embodiments may be arbitrarily combined with each other without conflict.

Related technologies often use a third-party tool to simply encrypt data for data protection or to authenticate users by introducing a secure authentication (Kerberos) component.

In the first related method, Hive's permission control process is unchanged. Before the data is imported into the Hive table, the original data is encrypted by a third-party encryption tool, and the encrypted data is imported into the Hive table, and the query is received. After the request, the data is exported from the Hive table, and the exported data is manually decrypted.

The first related method can only solve the problem in a limited way; because the data is imported and exported repeatedly, the operation is cumbersome and time consuming; the data is encrypted by deploying a third-party encryption tool outside the Hive, which increases the complexity of the system, and After the data is encrypted, the data length is generally increased. When the data of the changed length is imported into the Hive, the import efficiency of the Hive system is reduced. Because the data is encrypted before entering Hive, Hive's MapReduce processing function cannot be borrowed. Data encryption can only be performed according to a specified method (usually a table). It is not possible to flexibly select an encrypted object, such as selecting one or several tables, columns, and Partition or bucket, etc.

In the second related method, a third-party network authentication protocol (Kerberos) component is introduced in the Hive privilege control, and the Kerberos component of the third party can be directly accessed in the privilege control module of the Hive, as part of the privilege control module, through Kerberos The component authenticates the user and prevents malicious users from forging the information of the stored user.

In the second related method, the cost of deploying third-party Kerberos components is relatively high, and it is very complicated. The steps of generating certificates and configuration for Kerberos components are quite cumbersome. The first configuration steps may not be too cumbersome, but the user rights are modified and the machine is To reduce capacity and capacity, you need to regenerate the certificate, distribute the certificate, and restart the system. Moreover, the Kerberos downtime may lead to the risk that the entire cluster consisting of multiple nodes cannot be serviced. The configuration of Kerberos itself is also complicated, which leads to the performance degradation of the Hadoop cluster running jobs. Therefore, Kerberos is rarely used in big data. In addition, the second related method cannot solve the authorization level problem, and this method cannot protect the data of the underlying HDFS.

In summary, the data protection method in the related art has a poor security mechanism, is cumbersome to operate, and is time consuming, and cannot perform flexible operations on tables, columns, partitions, and buckets, and the degree of confidentiality is not high, and the performance of Hadoop cluster running operations is low. And the problem of higher deployment costs.

First embodiment

Embodiments of the present disclosure provide a data protection method in a data warehouse.

In the method, a user request input by a user is received, where the user request carries a user Identity information and an operation request, the operation request including one or more of a table level operation request, a column level operation request, a partition level operation request, and a bucket level operation request; determining the user identity information and the location Whether the operation request is legal; if the identity information and the operation request are both legal, determining the operation type of the operation request; and if the operation request is an unauthorized operation, adding to the data warehouse Data or existing data is encrypted.

Therefore, the data protection method in the data warehouse of the embodiment of the present disclosure can protect the data in the data warehouse by performing access control on the data warehouse user and permission control at the column, partition, or bucket level to encrypt the user data. .

As shown in Figure 1, the method includes the following steps.

In step 110, a user request entered by the user is received.

The user request carries user identity information and an operation request, and the operation request includes one or more of a table level operation request, a column level operation request, a partition level operation request, and a bucket level operation request. Kind.

When a user needs to access the Hive system for specific data operations, the user inputs a user request to the Hive system. In order to facilitate checking the identity information of the user and determining whether the user is a legitimate user, the user request may carry the identity information of the user, such as the user name and the Internet Protocol (IP) address of the user accessing the Hive system. .

In step 120, it is determined whether the user identity information and the operation request are legal.

Optionally, step 120 includes:

In the embodiment of the present disclosure, a whitelist is stored in the Hive system, and the whitelist stores identity information of a user having data access rights and a user having data access rights, such as a user name and a user accessing the Hive system. IP address, etc. When receiving the user request, the whitelist is read to determine whether the user identity information carried in the user request is legal, that is, whether the user is a user recorded in the whitelist, and if so, the user identity information is legal, and the user is allowed to The user accesses the Hive system. If not, the user is denied access to the Hive system.

The Hive system also stores preset operation rights for legitimate users to operate on data in the Hive system. For example, the default operational authority that each legitimate user has is stored in the operation permission table. After determining that the user accessing the Hive system belongs to the legal user, the read operation permission table determines whether the permission to execute the operation request is the preset operation authority of the user, that is, whether the user has a table corresponding to the operation request. Operation authority of a column, partition, or bucket (for example, determining whether the user has permission to create a table), and if so, the operation request is legal, and if not, the operation request is refused, and the process ends.

In step 130, if the identity information and the operation request are both legal, the operation type of the operation request is determined.

The operation type of the operation request is divided into an authorization operation and an unauthorized operation. The authorization operation may be that a legitimate user of the Hive system can authorize other users to access a column, a partition, or a bucket of the legitimate user. Unauthorized operations can include creating table operations, data import operations, modifying table operations, encrypting existing data, and data query operations.

When the operation request is an authorization operation, it may be determined whether one or more of the table, the column, the partition, and the bucket that the user carried in the authorization operation authorizes other users to operate belong to the user; if yes, execute the Authorization operation, if not, cancel the authorization operation. Embodiments of the present disclosure provide flexible authorization based on table, column, partition, and bucket levels, allowing Hive users to authorize other users to access a column, a partition, or a bucket.

When the operation request is an unauthorized operation, step 140 may be performed.

In step 140, if the operation request is an unauthorized operation, the new data or the existing data in the data warehouse is encrypted.

Among them, the process of encrypting new data in the data warehouse is included in the process of creating tables and data import. Optionally, encrypting new data in the data warehouse includes:

Obtaining the create table operation, and creating a table according to the structural information of the table carried in the create table operation;

Determining whether the first encryption configuration information is carried in the creation table operation, where the first encryption configuration information includes one or more of a name of a table to be encrypted, a name of a column, a name of a partition, and a name of a bucket. Kind, encryption algorithm and decryption algorithm;

If the creation table operation carries the first encryption configuration information, the first encryption configuration letter is The information is stored in the encrypted information table;

Obtaining the data import operation, wherein the data import operation carries a name of a first target table that needs to import data, a name of a first target column, a name of a first target partition, and a name of a first target bucket One or more, the first target table includes one or more tables in the created table, the first target column includes one or more columns of the created table, and the first target partition includes the created One or more partitions of the table, the first target bucket including one or more buckets of the created table;

For example, you can create a table X and encrypt the data in columns A and B located in table X. An empty table X with only a structure can be created according to the structural information of the table carried in the table creation operation, for example, the field name and the data type. Since there is data to be encrypted in the table X, the encryption table configuration information is carried in the table creation operation, and the encryption configuration information may include the name of the column A and the name of the column B to be encrypted, the encryption algorithm, and the decryption. algorithm.

For example, the name of column A is id, the name of column B is name, a table X is created by using SQL statement, and the data in column A and column B located in table X is encrypted, which can be as follows:

Create table encode_test(id INT,name STRING)ROW FORMAT SERDE

'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'

WITH SERDEPROPERTIES('column.encode.columns'='id,name',

'column.encode.classname'='com.zte.encode.DESRewriter')

STORED AS TEXTFILE;

Where column.encode.columns indicates the encrypted column, if it is empty, it means that it will be in the table. For all column encryption, the above example requires two columns to be encrypted: id and name; column.encode.classname indicates the encryption algorithm (class library), and the encryption algorithm can be Data Encryption Standard (DES). ). In the embodiment of the present disclosure, an open plug-in architecture may be adopted, that is, the Hive system includes an encryptor class library and a decryptor class library. Therefore, the encryptor corresponding to the encryption algorithm may be called to encrypt the data, and the call corresponds to the decryption algorithm. The decryptor decrypts the data and it is convenient to replace the encryption algorithm and the decryption algorithm.

In order to facilitate encrypting data that needs to be encrypted at the time of data import, the encrypted configuration information corresponding to Table X can be stored in the encrypted information table. The encryption information table includes an encryption algorithm used when encrypting based on a table, a column, a partition, or a bucket.

Once the table is created, you can import data into the empty table you created. For example, if you import data into column A in table X, the name of column A will be carried in the data import operation. The encryption algorithm corresponding to the A column is read from the encryption information table, and the data to be imported is encrypted by the encryption algorithm, and the encrypted data is written into the column A of the table X.

Since the data file corresponding to the data in the Hive is stored on the HDFS, the table X is created, that is, an empty directory corresponding to the table X is created in the HDFS, and the structure information (field name, field type, etc.) of the table X is created. , also known as dictionary information, stored in a physical database (such as MySQL, PostgreSQL, etc.).

The process of encrypting existing data in the data warehouse can be encrypted by modifying the table, or by creating a temporary table.

In the first method, the existing data in the data warehouse is encrypted by modifying the table, which may include:

Obtaining the modification table operation;

The encrypted configuration information is used as the carrying information of the modified table operation. When the data of a table, a column, a partition, or a bucket is encrypted, the table, the column, the partition, and the bucket may be carried in the modify table operation. At least one of the names, encryption algorithms, and decryption algorithms. When performing the modify table operation, the encryptor corresponding to the encryption algorithm may be invoked to encrypt the table, column, partition, or bucket that needs to be encrypted. Modify the encryption algorithm used in encryption to modify the encryption algorithm of the corresponding table, column, partition or bucket to the required algorithm. For example, change the encryption algorithm of partition DS=’2015’ to DES. The SQL statement is as follows:

ALTER TABLE C SET PARTITION (DS = '2015') 'partition.encode.classname' = 'com.zte.encode.DESRewriter'. The encryption configuration information carried in the modification table operation may be saved in the encryption information table.

In the second mode, the temporary table mode is created, and the existing data in the data warehouse is encrypted, which may include:

Acquiring the encryption operation on the existing data, where the encryption operation on the existing data carries the name of the to-be-processed table and the third encryption configuration information, where the third encryption configuration information includes the third target that needs to be encrypted. One or more of a name of a table, a name of a third target column, a name of a third target partition, and a name of a third target bucket, an encryption algorithm, and a decryption algorithm, the third target table including the to-be-processed table One or more of the tables, the third target column comprising one or more columns of the table to be processed, the third target partition comprising one or more partitions of the table to be processed, the third The target bucket includes one or more buckets of the to-be-processed table;

Creating a temporary table having the same structure as the pending table;

Correspondingly writing the query data into the temporary table;

Delete the temporary table.

The coverage method is to clear the contents of the original file and write new content.

Optionally, the acquiring the data query operation, and querying, in the third target table, the third target column, the third target partition, and the third target bucket according to the data query operation Data in one or more of the following, obtaining query data, including:

Acquiring the data query operation, wherein the data query operation carries the name of the third target table, the name of the third target column, the name of the third target partition, and the third target bucket One or more of the names;

Determining whether there is encrypted data in one or more of the third target table, the third target column, the third target partition, and the third target bucket;

For example, encrypting existing data in the M partition, the N partition, and the H partition in the table Y includes: creating a temporary table L having the same structure as the table Y, that is, creating a same as the table Y data dictionary, but not encrypting The temporary table L; and the data in the M partition, the N partition, and the H partition are read out from the HDFS in a data query manner. Among them, the data dictionary is a directory of record database and application metadata that the user can access.

When querying the table Y, the names of the M partition, the N partition, and the H partition that need to be queried can be obtained from the data query operation; and the data in the M partition, the N partition, and the H partition are read from the HDFS. Where the encrypted data may exist in the read data, the encrypted data may be decrypted before the read data is written into the temporary table L. When decrypting, the decryption algorithm corresponding to the table, column, partition or bucket where the encrypted data is located may be searched in the encrypted information table, and when the corresponding decryption algorithm is found, the decryptor pair corresponding to the decryption algorithm is applied to be encrypted. The data is decrypted to obtain the queried data.

After obtaining the data of the queried M partition, the N partition, and the H partition, the data is correspondingly written into the temporary table L. Since the temporary table L has the same structure as the table Y, the same partition exists in the temporary table L. Therefore, the queried data is correspondingly written into the M partition, the N partition, and the H partition in the temporary table L.

At this time, the data in the M partition, the N partition, and the H partition in the temporary table L may be encrypted by using an encryption algorithm carried in the existing data encryption operation. After encryption, the encrypted data in the M partition, the N partition, and the H partition are overwritten into the M partition, the N partition, and the H partition in the table Y, and the previously created temporary table L is deleted. In order to facilitate subsequent data operations on the data in the M partition, the N partition, and the H partition in the table Y, the encryption algorithm and the decryption algorithm corresponding to the M partition, the N partition, and the H partition may be stored in the encrypted information table.

After the data in the Hive system is encrypted by the above method, when the subsequent user performs data query, the name of the table to be queried, the name of the column, the name of the partition, and the name of the bucket may be obtained from the data query operation. One or more; reading data from one or more of the tables, columns, partitions, and buckets to be queried from HDFS; and looking up and reading the table, column, and partition from the encrypted information table And a decryption algorithm corresponding to one or more of the buckets, if the corresponding decryption algorithm is found, decrypting is performed by using the found decryption algorithm, and the decrypted data is returned to the user, if the corresponding decryption algorithm is not found. , the read data can be returned to the user.

In the Hive system, the encrypted information table may protect the data in the encrypted information table in the form of a hidden table in the metadata, and may also protect the data in the encrypted information table by encrypting the encrypted information table. After the first encrypted configuration information, the second encrypted configuration information, or the third encrypted configuration information is saved in the encrypted information table, the data in the encrypted information table may be encrypted by using a preset encryption algorithm.

The operation permission table and whitelist are also part of the metadata. Encrypting the operation permission table and the whitelist also protects the security of the operation permission table and the whitelist.

You can set the configuration parameter hive.metastore.encode.class (metadata encryption class and decryption class) to the specified encryption algorithm and decryption algorithm, such as Advanced Encryption Standard (AES) encryption algorithm and AES decryption algorithm: com.zte.encode.AESRewriter. The metastore is a metadata store. Before the metadata table (ie, the encryption information table, the operation permission table, and the whitelist) is written to the physical library, an encryption algorithm is used to encrypt all the contents of the table field. When the metadata is read from the physical database, the decryption algorithm is called to decrypt all the contents of the table field.

In the embodiment of the present disclosure, if the Hive system super administrator does not configure the access control, the user's access control cannot be performed at this time, and the users cannot mutually authorize each other, but the data can be encrypted and protected. When encrypting data queries, data decryption is only displayed when the owner of the data (the owner of the table, column, partition, or bucket data) is queried, and the non-data owner can only When you see the encrypted data, you can't see the decrypted data.

The large-scale data processing of the Hive system is implemented when the programming model (MapReduce) task runs. When MapReduce runs, the intermediate process produces some temporary data. Therefore, in some cases, temporary data generated by the intermediate process of MapReduce can be protected. During the execution of MapReduce, some data may need to be temporarily written to HDFS or disk. This data is called an intermediate process (such as the Map stage). Temporary data generated, temporary data is often a small fragment of unencrypted data. If you want to protect temporary data, you can also apply the encryption algorithm and decryption algorithm to the intermediate process of MapReduce, and the intermediate process data to HDFS. Or when the disk writes data, the encryption algorithm is called to encrypt the write data, and when the intermediate process reads data from the HDFS or the disk, the decryption algorithm is called to decrypt the read data.

You can set the configuration parameter hive.intermediate.compression.codec (intermediate data encoding) to the specified encryption algorithm and decryption algorithm, such as AES encryption algorithm and AES decryption algorithm (com.zte.encode.AESRewriter), which can realize intermediate generation of data. Encryption protection.

Second embodiment

The embodiment of the present disclosure further provides a data protection device in a data warehouse. As shown in FIG. 2, the data protection device 20 in the data warehouse includes: a receiving module 21, a first determining module 23, a second determining module 25, and encryption. Module 27.

The receiving module 21 is configured to receive a user request input by the user, where the user request carries user identity information and an operation request, where the operation request includes a table level operation request, a column level operation request, and a partition level operation request. And one or more of the bucket level operation requests.

The first determining module 23 is configured to determine whether the user identity information and the operation request are legal.

The second determining module 25 is configured to determine, when the first determining module 23 determines that the identity information and the operation request are both legal, the operation type of the operation request, where the operation type includes an authorization operation and a non-operation Authorized operation.

The encryption module 27 is configured to encrypt the newly added data or the existing data in the data warehouse when the second determination module 25 determines that the operation request is an unauthorized operation.

Optionally, as shown in FIG. 3, the first determining module 23 includes: a first determining unit 231 and a second determining unit 232.

The first determining unit 231 is configured to determine the user identity information according to the user identity information. Whether the corresponding user exists in a pre-stored white list, and if so, the user identity information is legal.

The second determining unit 232 is configured to determine whether the authority to execute the operation request is a preset operation authority possessed by the user, and if so, the operation request is legal.

Optionally, the unauthorized operation includes: creating a table operation, a data import operation, modifying a table operation, encrypting an existing data, and a data query operation.

Optionally, as shown in FIG. 3, the encryption module 27 includes: a first obtaining unit 271, a third determining unit 272, a first storage unit 273, a second obtaining unit 274, a fourth determining unit 275, and a third acquiring. Unit 276, first encryption unit 277, and first write unit 278.

The first obtaining unit 271 is configured to acquire a create table operation, and create a table according to the structure information of the table carried in the create table operation.

The third determining unit 272 is configured to determine whether the first encryption configuration information is carried in the creation table operation, where the first encryption configuration information includes a name of a table to be encrypted, a name of a column, a name of a partition, and a bucket. One or more of the names, encryption algorithms, and decryption algorithms.

The first storage unit 273 is configured to store the first encryption configuration information in the encryption information table when the third determination unit 272 determines that the creation table operation carries the first encryption configuration information.

The second obtaining unit 274 is configured to acquire a data import operation, where the data import operation carries a name of the first target table that needs to import data, a name of the first target column, a name of the first target partition, and a first target. One or more of the names of the buckets, the first target table comprising one or more tables in the created table, the first target column comprising one or more columns of the created table, the first The target partition includes one or more partitions of the created table, the first target bucket including one or more buckets of the created table.

The fourth determining unit 275 is configured to determine that one or more of the first target table, the first target column, the first target partition, and the first target bucket are in the encrypted information table. Whether a corresponding encryption algorithm is stored.

The third obtaining unit 276 is configured to: when the fourth determining unit 275 determines one of the first target table, the first target column, the first target partition, and the first target bucket or Acquiring one of the first target table, the first target column, the first target partition, and the first target bucket when a plurality of corresponding encryption algorithms are stored in the encryption information table Or a variety of corresponding encryption algorithms.

The first encryption unit 277 is configured to acquire data to be imported into one or more of the first target table, the first target column, the first target partition, and the first target bucket, and utilize the acquired The encryption algorithm encrypts the data to be imported to obtain the first encrypted data.

The first writing unit 278 is configured to write the first encrypted data correspondingly to one of the first target table, the first target column, the first target partition, and the first target bucket Or a variety of.

Optionally, as shown in FIG. 3, the encryption module 27 further includes: a fourth obtaining unit 279, a fifth determining unit 2710, a second encrypting unit 2711, and a second storing unit 2712.

The fourth obtaining unit 279 is configured to acquire a modification table operation;

The fifth determining unit 2710 is configured to determine whether the second encryption configuration information is carried in the modification table operation, where the second encryption configuration information includes a name of the second target table that needs to be encrypted, a name of the second target column, and a second One or more of the name of the target partition and the name of the second target bucket, an encryption algorithm, and a decryption algorithm.

The second encryption unit 2711 is configured to use the encryption algorithm carried in the modification table operation to the second target when the second modification unit 2710 determines that the modification table operation carries the second encryption configuration information. Data in one or more of the table, the second target column, the second target partition, and the second target bucket is encrypted.

The second storage unit 2712 is configured to store the second encrypted configuration information in the encrypted information table.

Optionally, as shown in FIG. 3, the encryption module 27 further includes: a fifth obtaining unit 2713, a creating unit 2714, a querying unit 2715, a second writing unit 2716, a third encrypting unit 2717, and a third writing unit. 2718, a third storage unit 2719, and a deletion unit 2720.

The fifth obtaining unit 2713 is configured to obtain an encryption operation on the existing data, where the encryption operation on the existing data carries the name of the to-be-processed table and the third encrypted configuration information, where the third encrypted configuration information includes One or more of an encrypted third target table name, a third target column name, a third target partition name, and a third target bucket name, an encryption algorithm, and a decryption algorithm, the third target table including One or more tables in the to-be-processed table, the third target column includes one or more columns of the to-be-processed table, and the third target partition includes one or more partitions of the to-be-processed table The third target bucket includes one or more buckets of the to-be-processed table.

The creating unit 2714 is set to create a temporary table having the same structure as the to-be-processed table.

The query unit 2715 is configured to acquire a data query operation, and query one of the third target table, the third target column, the third target partition, and the third target bucket according to the data query operation or A variety of data, get query data.

The second writing unit 2716 is arranged to write the query data correspondingly into the temporary table.

The third encryption unit 2717 is configured to encrypt the query data written in the temporary table by using the encryption algorithm carried in the encryption operation on the existing data to obtain the second encrypted data.

The third writing unit 2718 is configured to write the second encrypted data into the to-be-processed table in an overlay manner.

The third storage unit 2719 is configured to store the third encrypted configuration information in the encrypted information table.

The deleting unit 2720 is set to delete the temporary table.

Optionally, the query unit 2715 includes: a first obtaining subunit, a query subunit, a determining subunit, a second acquiring subunit, and a decrypting subunit.

The first obtaining subunit is configured to acquire the data query operation, wherein the data query operation carries a name of the third target table, a name of the third target column, a name of the third target partition, and One or more of the names of the third target buckets.

The query subunit is configured to read one of the third target table, the third target column, the third target partition, and the third target bucket from the distributed file system according to the data query operation Data in one or more.

The determining subunit is configured to determine whether the encrypted data exists in one or more of the third target table, the third target column, the third target partition, and the third target bucket.

The second obtaining subunit is configured to: when the judgment result of the determining subunit is YES, acquire one or more of a table, a column, a partition, and a bucket in which the encrypted data is located from the encrypted information table. Corresponding decryption algorithm.

Optionally, as shown in FIG. 3, the apparatus further includes: an authorization module 29.

The authorization module 29 is configured to: when the second determining module determines that the operation request is an authorized operation, Determining whether one or more of the tables, columns, partitions, and buckets that the user carried in the authorization operation authorizes other users to operate belong to the user, and if so, performing the authorization operation, if not, canceling the Authorized operation.

Optionally, the encryption module 27 further includes a fourth encryption unit 2721 as shown in FIG. The fourth encryption unit 2721 is configured to: after the first storage unit stores the first encryption configuration information in the encryption information table, encrypt the data in the encrypted information table by using a preset encryption algorithm; After storing the second encryption configuration information in the encryption information table, the second storage unit encrypts the data in the encrypted information table by using a preset encryption algorithm; and the third storage unit encrypts the third encryption After the configuration information is stored in the encrypted information table, the data in the encrypted information table is encrypted by using a preset encryption algorithm.

Third embodiment

4 is a structural block diagram of a data protection device in a data warehouse according to a third embodiment of the present disclosure. Data protection devices include:

The access control module 431 is configured to provide a whitelist function. The whitelist function of the access control module 431 is invoked by the rights control module 43. The users in the whitelist are allowed to access the Hive system, and the users in the whitelist are not allowed. It is not allowed to access the Hive system. The whitelist is configured by the super administrator of the Hive system according to the specified interface. It can be stored in the metadata module 41 in the form of a configuration table (whitelist) or in the form of a configuration file on the local hard disk. The access control module 431 is a plug-in component, and the user can inherit the specified interface and develop a customized whitelist function.

The privilege control module 43 is configured to provide a Hive system super administrator function on the basis of the Hive original privilege control module, invoke the whitelist function provided by the access control module 431, perform access control on the Hive system, and provide column-based, Flexible authorization at the partition and bucket level allows Hive users to authorize other users to access a column, a partition, or a bucket.

The statement parsing module 44 is configured to parse the SQL statement. The encrypted information of the present disclosure is defined when the table is created or the SQL statement of the table is modified. Therefore, based on the original Hive statement parsing module, the parsing of the encrypted information is added, and the encrypted information is written as a table to the metadata. Module 41. The encrypted information can include whether the table is encrypted, the name of the encrypted column, partition and bucket, and the name of the encryptor.

The serialization module 45 is arranged to provide functionality to write data into the HDFS 47. In the process of serialization, the data length tends to expand. In order to reduce the overhead of MapReduce network transmission, data encryption can be performed when data is written to HDFS 47.

The deserialization module 46 is arranged to provide a function for reading the table file from the HDFS 47. Since the data decryption length is reduced, in order to reduce the overhead of the MapReduce network transmission, the data can be decrypted during the deserialization process.

The HDFS 47 is set to store data files corresponding to data in the Hive system. The data in the HDFS 47 can be read by the deserialization module 46, and the data is written into the HDFS 47 by the serialization module 45.

The metadata module 41 is configured to define an encryption information table, a white list table, and an operation authority table, and store the encrypted information of the table, the whitelist information, and the preset operation authority of the user. The encrypted information of the table includes whether the table is encrypted, the encrypted column, the name of the partition bucket, and the name of the encryptor. In order to minimize the impact on the user's SQL program, that is, the user's existing SQL program is not modified, so that the user executes the same SQL program in the case of encryption and non-encryption, and the disclosure defines the encryption information table and the white list. When the table and the operation permission table are used, the encrypted information is persisted to the metadata module 41 as part of the table data dictionary. When there is data operation, any changes are to be made, and the serialization module 45 or the deserialization module 46 is encrypted according to the encryption. The information determines whether the data needs to be encrypted or decrypted. The whitelist table stores the user name and the IP address of the user accessing the Hive system.

The data encryption and decryption module 42 is a separate plug-in component that is respectively called by the serialization module 45 and the deserialization module 46 to provide a data encryption algorithm and a decryption algorithm to implement data encryption and decryption functions. The present disclosure adopts an AES encryption algorithm and an AES decryption algorithm. AES is also called Rijndael encryption in cryptography, and is a block encryption standard adopted by the US federal government. AES can replace DES. AES has become one of the most popular algorithms for symmetric key encryption. AES features fast encryption, compact coding and high security. At the same time, the encrypted data generated by the encryption algorithm is only related to the original data and the encryption key, that is, as long as the encrypted original data and the key are determined, in any case, the generated encrypted data is the same. The same is true for decryption. As long as the key is obtained, the encrypted data can be decrypted, and the algorithm can be used in a multi-host and multi-node computing environment such as MapReduce. At the same time, since the present disclosure adopts a plug-in architecture, a user can develop a custom encryption or decryption plugin by inheriting the interface defined by the present disclosure.

The key configuration module 421 is a sub-module of the data encryption and decryption module 42 and can be used to synthesize the encrypted key and the decrypted key. In the encryption process, the Hive system allocates a 32-bit length for each user for encryption. Encryption key (key), the assigned key and the user-defined key synthesize the user's encryption key, and write through the Java Database Connectivity (JDBC) interface The metadata module 41 is entered.

Based on the structural block diagram of FIG. 4, when there is a user access, the plurality of modules shown in FIG. 4 are executed in accordance with the flowchart shown in FIG.

In step 510, the user requests that the user enter a user request.

In step 520, the user whitelist is read by the metadata module 41. Because the whitelist stores the identity information of the user with the legal access authority, the whitelist can be read from the metadata module 41 in order to facilitate the determination of the identity information of the user who inputs the user request.

In step 530, it is determined whether the user is in the white list. The privilege control module 43 invokes the whitelist function provided by the access control module 431 to determine whether the user is in the whitelist. If the user is in the whitelist, the user is allowed to access the Hive system, and step 540 is performed. Within the list, the user is not allowed to access the Hive system. Step 5130 is executed to complete the process. Therefore, only the user authenticated by the rights control module 43 is allowed to access the Hive system and participate in subsequent operations.

In step 540, the rights control module 43 obtains the user rights and compares the obtained user rights with the rights required in the operation. The authority control module 43 reads the operation authority control table from the metadata module 41.

In step 550, it is determined whether the operation request in the user request is an authorization operation. If the operation request is an authorization operation, step 560 is performed. If the operation request is not an authorization operation, step 570 is performed.

In step 560, if the authorized table, column, partition, or bucket belongs to the user, the authorization is successful. If the authorized table, column, partition, or bucket does not belong to the user, the authorization is not successfully performed, and the process ends.

In step 570, it is determined whether the user has the corresponding operation authority. If yes, step 580 is executed. If not, step 5130 is performed.

In step 580, the type of operation of the user's operation request is determined. Among them, the operation types include creating a table, modifying a table, importing data, querying data, and encrypting and protecting existing data.

When the operation request is to create a table or modify the table, step 590 is performed, see the first branch flow of FIG.

When the operation request is data import, step 5100 is performed, see the second branch flow of FIG.

When the operation request is a data query, step 5110 is performed, see the third branch flow of FIG.

When the operation request is to encrypt and protect the existing data, step 5120 is performed, see the fourth branch flow of FIG.

As shown in FIG. 6, when the operation request is to create a table or modify the table, steps 5910 to 5950 are performed.

In step 5910, a table or a modified table statement is created. If you need to create an encrypted table, column, partition, or bucket, you can define an encrypted table, column, partition, or bucket in the data dictionary.

In step 5920, the data dictionary information for the table is written to the physical database by the metadata module 41.

In step 5930, it is determined whether the statement of the creation table or the modification table carries the encryption configuration information. If yes, step 5940 is performed, and if not, step 5950 is performed.

In step 5940, the encrypted configuration information is written by the metadata module 41 to the encrypted information table in the physical database.

In step S5950, the flow is ended.

As shown in FIG. 7, when the operation request is data import, steps 51010 to 51070 are performed.

In step 51010, a MapReduce process is executed.

In step 51020, the serialization module 45 is invoked to serialize the data to the HDFS 47.

In step 51030, the metadata table information is read by the metadata module 41. The metadata table information includes encrypted information.

In step 51040, it is determined whether the imported target table, column, partition or bucket is encrypted. If yes, step 51050 is performed, and if not, step 51060 is performed.

In step 51050, during the serialization process, the encryption algorithm provided by the encryption and decryption module 42 is invoked for data encryption. The serialization module 45 invokes the key configuration module 421 to synthesize a key, and the calling metadata module 41 acquires the name of the column, partition or bucket to be encrypted, the name of the encryption algorithm, and the name of the decryption algorithm, and loads the specified encryption algorithm and decryption algorithm. Data encryption is done based on the specified encrypted data (columns, partitions, or buckets to be encrypted) and the key.

In step 51060, data is written to HDFS 47 and MapReduce ends.

In step 51070, the process ends. After the data is written to HDFS 47, the process in this section ends.

As shown in FIG. 8, when the operation request is a data query, steps 51110 to 51180 are performed.

In step 51110, a MapReduce process is executed.

In step 51120, the deserialization module 46 is called to read data from the HDFS 47.

In step 51130, the metadata information is read by the metadata module 41. The metadata information includes encrypted table information.

In step 51140, it is determined whether the table, column, partition or bucket of the query is encrypted. If it is encrypted, step 51150 is performed. If there is no encryption, step 51160 is performed.

In step 51150, during the deserialization process, the decryption algorithm provided by the encryption and decryption module 42 is invoked for data decryption.

In step 51160, after the data is read, MapReduce ends.

In step 51170, the query data is returned to the user.

In step 51180, the flow ends. After the query data is returned to the user, the process ends.

As shown in FIG. 9, when the operation request is to encrypt and protect the existing data, steps 5121 to 51211 are performed.

In step 5121, a query is performed on the source table (columns, partitions, or buckets).

In step 5122, the data to be queried is obtained.

In step 5123, a common temporary table with the same homologous table data dictionary but no encryption is created.

In step 5124, the data queried above is imported into the temporary table.

In step 5125, MapReduce is executed.

In step 5126, at the end of the MapReduce flow, the serialization module 45 is invoked to serialize the data to the HDFS 47.

In step 5127, it is determined whether some tables, columns, partitions, or buckets of the source table need to be encrypted. If necessary, step 5128 is performed. If not, step 5129 is performed.

In step 5128, during the serialization process, the encryption algorithm provided by the encryption and decryption module 42 is invoked for data encryption.

In step 5129, the data is written to HDFS 47 and the MapReduce process ends. The encrypted data is written to the source table in an overwrite manner.

In step S51210, the ordinary temporary table is deleted to save the storage space of the system.

In step S51211, the flow is ended. After the normal temporary table is deleted, the part of the process ends.

The order of creating the temporary table and the querying step of the table are not limited thereto.

In the block diagram shown in FIG. 4, encryption is performed due to the presence of the data encryption and decryption module 42. Or the decryption algorithm has been loaded into the system. The encryption function and the decryption function can be defined by a User Defined Functions (UDF) without having to define the data dictionary, so that the encryption information can be obtained without parsing by the statement parsing module 44, and the encrypted information can be stored in the metabase. the process of.

For example: select a, encode(b) from tbl, encrypt the contents of the b field of the table tbl through the encode interface in the encryption algorithm, and import the encrypted result a and encode(b) into the predefined table tbl_encrypt, the table Tbl_encrypt can be used as a place to store data after encryption. When the query request is received, the decryption function decode is called, such as: select a, decode(c) from tbl_encrypt, and the content of c is encode(b). This way does not need to be in the data like creating a table or modifying a table. The dictionary defines the encrypted table, column, partition or bucket. It can modify the original SQL statement, increase the call to the user-defined function, and artificially ensure that the encrypted table, column, partition or bucket is exactly the same when decrypted.

In summary, the present disclosure protects Hive data through user rights control and data encryption, and implements access control and column, partition, or bucket level authority control for users of the Hive system, and combines user data based on tables, Encryption and decryption at the column, partition or bucket level, protecting the data in the Hive data warehouse, reducing the security mechanism in the related technology, the operation is cumbersome and time-consuming, and the table, column, partition and bucket cannot be flexibly encrypted. The phenomenon of low confidentiality, inefficiency and high deployment costs.

Fourth embodiment

Embodiments of the present disclosure also provide a non-transitory computer readable storage medium storing computer executable instructions arranged to perform any of the methods described above.

Fifth embodiment

Embodiments of the present disclosure provide a hardware structure diagram of an electronic device. Referring to FIG. 10, the electronic device includes:

At least one processor 100, which is exemplified by a processor 100 in FIG. 10; and a memory 101, may further include a communication interface 102 and a bus 103. The processor 100, the communication interface 102, and the memory 101 can complete communication with each other through the bus 103. Communication interface 102 can be used for information transmission. The processor 100 can call logic instructions in the memory 101 to perform the above method.

In addition, the logic instructions in the memory 101 described above can be implemented in the form of software functional units. And when sold or used as a stand-alone product, it can be stored on a computer readable storage medium.

The memory 101 is a computer readable storage medium, and can be used to store a software program, a computer executable program, a program instruction or a module corresponding to the method in the embodiment of the present disclosure. The processor 100 performs functional applications and data processing by running software programs, instructions or modules stored in the memory 101.

The memory 101 may include a storage program area and an storage data area, wherein the storage program area may store an operating system, an application required for at least one function; the storage data area may store data created according to usage of the terminal, and the like. Further, the memory 101 may include a high speed random access memory, and may also include a nonvolatile memory.

The technical solution of the present disclosure may be embodied in the form of a software product stored in a storage medium, including one or more instructions for causing a computer device (which may be a personal computer, a server, a network device, etc.) Performing all or part of the steps of the method of the embodiments of the present disclosure. The foregoing storage medium may be a non-transitory storage medium, including: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random storage memory (RAM), a magnetic disk, or an optical disk. A medium that can store program code, or a transitory storage medium.

Industrial applicability

The data protection method and device in the data warehouse provided by the disclosure can perform user access control on the data warehouse and permission control at the column, partition or bucket level to encrypt the user data, and can perform data on the data warehouse. protection.

Claims

A data protection method in a data warehouse, comprising:

Receiving a user request, where the user request carries user identity information and an operation request, where the operation request includes a table level operation request, a column level operation request, a partition level operation request, and a bucket level operation request. One or more;

Determining whether the user identity information and the operation request are legal;

If the identity information and the operation request are both legal, determining an operation type of the operation request, where the operation type includes an authorization operation and an unauthorized operation;

If the operation request is the unauthorized operation, the new data or the existing data in the data warehouse is encrypted.
The method of claim 1, wherein the determining the user identity information and the operation request are legal, comprising:

Determining, according to the user identity information, whether the user corresponding to the user identity information exists in a pre-stored white list, and if yes, the user identity information is legal;

Determining whether the permission to execute the operation request is a preset operation authority of the user, and if so, the operation request is legal.
The method of claim 2, wherein the unauthorized operation comprises: creating a table operation, a data import operation, modifying a table operation, encrypting an existing data, and a data query operation.
The method of claim 3, wherein encrypting the new data in the data repository comprises:

Obtaining the create table operation, and creating a table according to the structural information of the table carried in the create table operation;

Determining whether the first encryption configuration information is carried in the creation table operation, where the first encryption configuration information includes one of a name of a table to be encrypted, a name of a column, a name of a partition, and a name of a bucket. Multiple, encryption algorithms and decryption algorithms;

If the creation of the table operation carries the first encryption configuration information, storing the first encryption configuration information in the encryption information table;

Obtaining the data import operation, where the data import operation carries one of a name of a first target table, a name of a first target column, a name of a first target partition, and a name of a first target bucket that need to import data. Or a plurality, the first target table includes one or more tables in the created table, the first target column includes one or more columns of the created table, and the first target partition includes the created table One or more partitions, the first target bucket including one or more buckets of the created table;

Determining whether one or more of the first target table, the first target column, the first target partition, and the first target bucket store a corresponding encryption algorithm in the encryption information table;

If yes, obtaining an encryption algorithm corresponding to one or more of the first target table, the first target column, the first target partition, and the first target bucket;

Obtaining data to be imported into one or more of the first target table, the first target column, the first target partition, and the first target bucket, and using the obtained encryption algorithm to compare data to be imported Encrypting to obtain the first encrypted data;

And correspondingly writing the first encrypted data into one or more of the first target table, the first target column, the first target partition, and the first target bucket.
The method of claim 4, wherein encrypting the existing data in the data repository comprises:

Obtaining the modification table operation;

Determining whether the modification table operation carries the second encryption configuration information, where the second encryption configuration information includes a name of the second target table that needs to be encrypted, a name of the second target column, a name of the second target partition, and a second One or more of the names of the target buckets, an encryption algorithm, and a decryption algorithm;

And if the modification table operation carries the second encryption configuration information, using the encryption algorithm carried in the modification table operation, the second target table, the second target column, the second target partition, and the Encrypting data in one or more of the second target buckets;

The second encrypted configuration information is stored in the encrypted information table.
The method of claim 4 wherein encrypting said existing data in said data repository comprises:

Acquiring the encryption operation on the existing data, where the encryption operation on the existing data carries the name of the table to be processed and the third encryption configuration information, where the third encryption configuration information includes the third target table that needs to be encrypted. One or more of a name, a name of a third target column, a name of a third target partition, and a name of a third target bucket, an encryption algorithm, and a decryption algorithm, the third target table including in the to-be-processed table One or more tables, the third target column including one or more columns of the pending table, the third target partition including one or more partitions of the pending table, the third target bucket Include one or more buckets of the pending table;

Creating a temporary table having the same structure as the pending table;

Obtaining the data query operation, and querying one or more of the third target table, the third target column, the third target partition, and the third target bucket according to the data query operation Data, obtain query data;

Correspondingly writing the query data into the temporary table;

Encrypting the query data written in the temporary table by using the encryption algorithm carried in the encryption operation of the existing data to obtain the second encrypted data;

Writing the second encrypted data into the to-be-processed table in an overlay manner;

Storing the third encrypted configuration information in the encrypted information table;

Delete the temporary table.
The method of claim 6, wherein the obtaining the data query operation and querying the third target table, the third target column, the third target partition, and the location according to the data query operation Data in one or more of the third target buckets, obtaining query data, including:

Acquiring the data query operation, where the data query operation carries the name of the third target table, the name of the third target column, the name of the third target partition, and the name of the third target bucket One or more

Reading one or more of the third target table, the third target column, the third target partition, and the third target bucket from a distributed file system according to the data query operation The data;

Determining whether there is encrypted data in one or more of the third target table, the third target column, the third target partition, and the third target bucket;

And if so, obtaining, from the encrypted information table, a decryption algorithm corresponding to one or more of a table, a column, a partition, and a bucket in which the encrypted data is located;

Using the decryption algorithm, the encrypted data is decrypted to obtain the query data.
The method of claim 1, after determining the type of operation of the operation request, the method further comprises:

If the operation request is an authorization operation, determining whether one or more of the table, column, partition, and bucket in which the user corresponding to the user identity information carried in the authorization operation authorizes other users to operate belongs to the User;

If yes, the authorization operation is performed, and if not, the authorization operation is cancelled.
The method of any one of claims 4 to 6, further comprising:

The data in the encrypted information table is encrypted by using a preset encryption algorithm.
A data protection device in a data warehouse, comprising:

a receiving module, configured to receive a user request, where the user request carries a user identity Information and operation requests, the operation request including one or more of a table level operation request, a column level operation request, a partition level operation request, and a bucket level operation request;

a first determining module, configured to determine whether the user identity information and the operation request are legal;

a second determining module, configured to determine, when the first determining module determines that the identity information and the operation request are both legal, the operation type of the operation request, where the operation type includes an authorized operation and an unauthorized Operation;

The encryption module is configured to encrypt new data or existing data in the data warehouse when the second determining module determines that the operation request is an unauthorized operation.
The apparatus of claim 10, wherein the first determining module comprises:

The first determining unit is configured to determine, according to the user identity information, whether the user corresponding to the user identity information exists in a pre-stored white list, and if yes, the user identity information is legal;

The second determining unit is configured to determine whether the permission to execute the operation request is a preset operation authority of the user, and if yes, the operation request is legal.
The apparatus of claim 11, wherein the unauthorized operation comprises: creating a table operation, a data import operation, modifying a table operation, encrypting an existing data, and a data query operation.
The apparatus of claim 12 wherein said encryption module comprises:

a first obtaining unit, configured to acquire the create table operation, and create a table according to the structural information of the table carried in the create table operation;

The third determining unit is configured to determine whether the first encryption configuration information is carried in the creation table operation, where the first encryption configuration information includes a name of a table to be encrypted, a name of a column, a name of a partition, and a name of a bucket. One or more, an encryption algorithm and a decryption algorithm;

a first storage unit, configured to be carried in the operation of determining the creation table by the third determining unit When the first encryption configuration information is available, the first encryption configuration information is stored in the encryption information table;

a second obtaining unit, configured to acquire the data import operation, where the data import operation carries a name of a first target table that needs to import data, a name of the first target column, a name of the first target partition, and a first target One or more of the names of the buckets, the first target table comprising one or more tables in the created table, the first target column comprising one or more columns of the created table, the first The target partition includes one or more partitions of the created table, the first target bucket including one or more buckets of the created table;

a fourth determining unit, configured to determine whether one or more of the first target table, the first target column, the first target partition, and the first target bucket are in the encryption information table Storing a corresponding encryption algorithm;

a third acquiring unit, configured to: when the fourth determining unit determines one or more of the first target table, the first target column, the first target partition, and the first target bucket Acquiring one or more of the first target table, the first target column, the first target partition, and the first target bucket when a corresponding encryption algorithm is stored in the encryption information table Corresponding encryption algorithm;

a first encryption unit, configured to acquire data to be imported into one or more of the first target table, the first target column, the first target partition, and the first target bucket, and utilize the acquired The encryption algorithm encrypts the data to be imported to obtain the first encrypted data;

a first writing unit, configured to write the first encrypted data correspondingly to one of the first target table, the first target column, the first target partition, and the first target bucket Or a variety of.
The apparatus of claim 13 wherein said encryption module comprises:

a fourth obtaining unit, configured to acquire the modified table operation;

a fifth determining unit, configured to determine whether the second encryption configuration information is carried in the modification table operation, where the second encryption configuration information includes a name of the second target table that needs to be encrypted, and a second target column One or more of a name, a name of a second target partition, and a name of a second target bucket, and an encryption or decryption algorithm;

a second encryption unit, configured to: when the second determination unit carries the second encryption configuration information in the operation of the modification table, using the encryption algorithm carried in the modification table operation on the second target table, Encrypting data in one or more of the second target column, the second target partition, and the second target bucket;

The second storage unit is configured to store the second encrypted configuration information in the encrypted information table.
The apparatus of claim 13 wherein said encryption module comprises:

The fifth obtaining unit is configured to obtain the encryption operation on the existing data, where the encryption operation on the existing data carries the name of the to-be-processed table and the third encryption configuration information, where the third encryption configuration information includes One or more of a name of the encrypted third target table, a name of the third target column, a name of the third target partition, and a name of the third target bucket, and an encryption or decryption algorithm, the third target table including One or more tables in the to-be-processed table, the third target column includes one or more columns of the to-be-processed table, and the third target partition includes one or more partitions of the to-be-processed table The third target bucket includes one or more buckets of the to-be-processed table;

Creating a unit, set to create a temporary table having the same structure as the pending table;

a query unit, configured to acquire the data query operation, and query one of the third target table, the third target column, the third target partition, and the third target bucket according to the data query operation Data in one or more kinds, obtaining query data;

a second writing unit, configured to write the query data into the temporary table correspondingly;

The third encryption unit is configured to encrypt the query data written in the temporary table by using the encryption algorithm carried in the encryption operation on the existing data to obtain the second encrypted data;

a third writing unit, configured to write the second encrypted data into the to-be-received manner in an overlay manner In the table;

a third storage unit configured to store the third encrypted configuration information in the encrypted information table;

Delete the unit, set to delete the temporary table.
The apparatus of claim 15 wherein said querying unit comprises:

a first obtaining subunit, configured to acquire the data query operation, where the data query operation carries a name of the third target table, a name of the third target column, a name of the third target partition, and a location One or more of the names of the third target buckets;

Querying a subunit, configured to read from the distributed file system the third target table, the third target column, the third target partition, and the third target bucket according to the data query operation Data in one or more;

a determining subunit, configured to determine whether the encrypted data exists in one or more of the third target table, the third target column, the third target partition, and the third target bucket;

a second obtaining subunit, configured to determine one or more of the third target table, the third target column, the third target partition, and the third target bucket when the determining subunit determines And obtaining, in the encrypted information table, a decryption algorithm corresponding to one or more of a table, a column, a partition, and a bucket in which the encrypted data is located; and

The decryption subunit is configured to decrypt the encrypted data using the decryption algorithm to obtain the query data.
The device of claim 10, the device further comprising:

And an authorization module, configured to: when the second determining module determines that the operation request is an authorization operation, determining that the user corresponding to the user identity information carried in the authorization operation authorizes other users to operate the table, column, partition, and bucket Whether one or more of the users belong to the user, and if so, perform the grant Right operation, if not, cancel the authorization operation.
The device according to any one of claims 13 to 15, wherein the encryption module further comprises:

The fourth encryption unit is configured to encrypt the data in the encrypted information table by using a preset encryption algorithm.
A non-transitory computer readable storage medium storing computer executable instructions arranged to perform the method of any of claims 1-9.