CN114896584B - A Hive data permission control proxy layer method and system - Google Patents

Abstract

The invention discloses a Hive data authority control agent layer method and a Hive data authority control agent layer system, which comprise the following steps: s1: hive data authority application; s2: resolving HQL; s3: HQL rewriting; s4: checking HQL authority; s5: HQL table, field blood margin analysis. The method and the system can meet the requirement of fine-grained data authority control. The method can meet the requirement of fine-grained data authority control in an enterprise, HQL analysis, rewriting and authority verification are realized on the basis of Hive prototypes, accuracy and stability are guaranteed to a certain extent, and the requirement of production on-line is met to a certain degree; the Java class Datacanthus authority and Datacanthus authority provided by the invention are both specifically realized for the Hive interface, and can realize data authentication at employee/user level; the method and the system do not need to invade a group big data cluster management platform during installation and deployment, and can meet the requirements of a large part of actual production scenes.

Description

A Hive data permission control proxy layer method and system

technical field

The invention relates to the technical field of network and data processing, in particular to a Hive data authority control agent layer method and system.

Background technique

With the wide application of technologies such as Internet open source technology and cloud computing, basic technologies such as performance and scalability are no longer the bottleneck of enterprise business development. Breaking through data silos, maximizing data intelligence, and data-driven business have become the core competitiveness of an enterprise's future development. At present, with the vigorous development of big data open source technologies and communities, enterprises usually choose a big data technology stack with "Hadoop" as the core, such as HDFS (Note: a distributed file storage system in the "Hadoop" ecosystem )/HBase (Note: a non-relational database system based on columnar storage in the "Hadoop" ecosystem) for distributed storage, Hive (Note: an offline query engine in the "Hadoop" ecosystem)/Spark (Note: " A memory-based distributed computing framework and offline computing engine in the Hadoop ecosystem) for offline data analysis, Flink (Note: a real-time computing engine capable of stream-batch integration in the "Hadoop" ecosystem) for real-time data analysis, Yarn (note: a general resource management system in the "Hadoop" ecosystem) for resource management and distributed task scheduling, Ranger/Sentry (note: two different data permission control frameworks and components in the "Hadoop" ecosystem) for big data Data access control for clusters. However, a problem that must be solved after the data of different departments of the enterprise is imported into the data center is the problem of data permission control.

At the same time, in order to integrate resources, enterprises usually have a dedicated big data and security department responsible for building and managing CDH (Cloudera's Distribution Including Apache Hadoop, CDH)/HDP (HortonWorks Data Platform, HDP)/EMR (Elastic MapReduce, EMR) ) as the representative of the big data cluster management platform, and other departments apply for Hive database, HDFS file directory and other resources to the group's big data cluster management platform as a tenant. The big data and security department allocates resource permissions to tenants through the Ranger/Sentry components that come with CDH/HDP/EMR, thereby realizing data isolation between tenants. This kind of data isolation between tenants is a coarse-grained data permission control scheme, which cannot meet the more fine-grained data permission control requirements within the department. B, as a common visitor, can only have viewing rights for specific tables. This is mainly because all employees in the department interact with the big data cluster through the same tenant. The identity authentication and data authentication of Ranger and Sentry can only be for the tenant, and cannot be further refined to different employees under the tenant.

Most of the Hive data permission schemes that are often proposed are still based on Ranger and Sentry. They create permission policies to Ranger Admin or create roles to Sentry and assign permissions to roles through the external interfaces provided by Ranger and Sentry. In addition, another part of the solution is to re-engineer the plugins of Ranger and Sentry to realize custom data permission control. Because Ranger and Sentry both use a plug-in method to control data permissions for big data components such as Hive and HDFS, that is, plug-ins need to be embedded in these big data components. These solutions are intrusive data permission control solutions. For the scenario where the big data cluster management platform is managed by the group in a unified manner, it is not allowed for the department to develop and deploy the permission plug-in to the group's big data cluster management platform through self-developed permission plug-ins. Therefore, in many real scenarios, the Hive data permission control agent layer solution that does not invade the big data cluster has become the only choice for implementing fine-grained data permission control within the enterprise department.

As mentioned above, most of the proposed Hive data permission schemes are based on Ranger and Sentry plug-in schemes. Through the external interface provided by Ranger and Sentry, they create permission policies to Ranger Admin (Note: Ranger's policy management center) or create roles to Sentry and assign permissions to roles, and then use Ranger and Sentry's native or re-transformed Hive The plugin pulls permission policies from Ranger Admin and Sentry to complete data permission authentication. These solutions all require the Hive plugins of Ranger and Sentry to be placed in the CDH/HDP/EMR Hive dependency package directory, and Hive must be configured and restarted to take effect. On the one hand, the Hive plug-in will invade HiveServer2 (Note: Hive's server) to execute the HQL (or HiveQL, a SQL dialect provided by Hive) process. The security of the Hive plug-in is usually not trusted for the group big data cluster management platform. . This is why in many realistic scenarios, especially the big data cluster management platform is managed and maintained by a specific department of the group and shared by other departments, Hive plug-ins provided by third parties are not allowed to be put into the big data cluster management platform . On the other hand, the installation and update of Hive plug-ins require Hive to be configured and restarted, which will lead to interruption of Hive services, resulting in the failure of Hive task execution results to return normally, resulting in production accidents. Due to these two reasons, the existing Hive data permission scheme can only be applied to the CDH/HDP/EMR scenario where the department independently deploys and maintains. Therefore, it is of great production value to realize the data authority agent layer scheme for Hive, an indispensable big data query engine in the data center. The so-called "data authority agent layer" refers to the provision of data authentication services by an agent outside the big data cluster management platform without invading the big data cluster management platform. Aiming at the realistic scenario that most big data cluster management platforms are centrally managed and controlled by the group, the present invention proposes a Hive data authority control agent layer scheme, which can meet the requirements of internal departments without using any plug-ins to invade the group big data cluster management platform. More fine-grained data permission control requirements.

SUMMARY OF THE INVENTION

In view of the problems existing in the prior art, the purpose of the present invention is to provide a Hive data authority control agent layer method and system, which can satisfy the needs of more fine-grained internal departments without using any plug-ins to invade the group big data cluster management platform. Data access control requirements.

In order to achieve the above object, the present invention provides a Hive data authority control proxy layer method, the method comprises the following steps:

S1: Hive data permission application; when the department's data permission approval service approves the employee's Hive data permission application, the data permission management center synchronously creates the corresponding data permission policy, which is stored in the table and field permission module and row filtering and field masking module , and update the mapping relationship between employees and data permission policies in user rights management;

S2: HQL parsing; before HQL parsing, it is necessary to use the tenant keytab that interacts with the group's big data cluster management platform for kerberos authentication, so as to achieve data isolation between tenants;

S3: HQL rewriting; SemanticAnalyzer performs row filtering and field desensitization rewriting of HQL through Hive's TableMask object in the process of parsing HQL;

S4: HQL authorization verification; based on QueryState, SemanticAnalyzer and HQL, call the Driver's static doAuthorization method to realize HQL data authorization verification;

S5: HQL table and field blood relationship analysis; after HQL authentication is passed, HQL table and field blood relationship analysis is performed.

Further, when an employee submits a Hive task, the JDBC-based HQL submission module sends the HQL and the employee's identity information to the Hive data authentication agency unit for authentication. The process of HQL authentication is the above-mentioned HQL analysis, HQL rewriting, and HQL authority. Checksum HQL table and field blood relationship analysis; if HQL authentication is passed, the JDBC-based HQL submission module submits the rewritten HQL to the group big data cluster management platform for execution.

Further, in step S2, after the kerberos authentication is passed, a HiveConf is created; the creation of the HiveConf relies on the Hadoop and Hive configuration files provided by the group's big data cluster management platform, and the configuration files include core-site.xml, hdfs-site.xml, mapred -site.xml, yarn-site.xml and hive-site.xml.

Further, in step S2, the HQL parsing includes the following sub-steps:

S201, use HiveConf to create a SessionState object, and set the userName of the SessionState object as the employee account submitting HQL;

S202, start the SessionState object, set the current database as the Hive database applied by the department on the big data cluster management platform, and initialize the transaction manager; after the SessionState object is created and started, it is valid and unique, and can communicate with Hadoop to submit distributed tasks. It can also connect to Hive's metadata database to query metadata information;

S203, create QueryState, Context, and ParseDriver objects in sequence; call the parse method of the ParseDriver object to parse the original HQL into an abstract syntax tree node; use the get method of Hive's SemanticAnalyzerFactory to generate the SemanticAnalyzer corresponding to QueryState and ASTNode;

S204, call the analyze method of SemanticAnalyzer to analyze the HQL.

Further, in step S3, the process of HQL rewriting includes the following sub-steps:

S301, traverse and parse the ASTNode of HQL to obtain table and field information;

S302, pull the row filtering and field masking permission policies of the table and field corresponding to the userName of SessionState from the data rights management center through DatablackHiveAuthorizer, and call the applyRowFilterAndColumnMasking method so that the TableMask object can correctly obtain the row filtering and field masking expressions;

S303, rewrite the Token stream of the original HQL according to the row filtering and field desensitization expressions, and save the Token stream of the latest HQL in the Context object.

Further, in step S4, if the HQL permission verification is successful, the method returns normally, otherwise, an exception of permission verification failure is thrown; the bottom layer of the driver's permission verification depends on the DatablackHiveAuthorizer class; the DatablackHiveAuthorizer class is a HiveAuthorizer for Hive provided by the present invention The implementation of the interface implements the checkPrivileges, applyRowFilterAndColumnMasking and needTransform methods; the driver's doAuthorization parses the HQL HiveOperationType, input and output HivePrivilegeObject and authentication context, and then calls the checkPrivileges method of the DatablackHiveAuthorizer; the checkPrivileges method pulls the user permission policy from the data rights management center, Parse the tables, fields and operation types involved in the input and output HivePrivilegeObject, and then match the user permission policy; if all input and output HivePrivilegeObject objects pass the permission verification, the method returns normally, otherwise, the permission verification failure exception is thrown.

Further, in step S5, the HQL table and field blood relationship analysis specifically includes the following steps:

S501, based on HiveConf, QueryState, SemanticAnalyzer and HQL, create QueryPlan and HookContext objects;

S502, call the run method of the Java class ColumnLineageAnalysis provided by the present invention, and return the blood relationship of tables and fields in HQL; ColumnLineageAnalysis is an inheritance class of LineageLogger of Hive, and rewrites the run method therein to return the blood relationship of tables and fields in HQL;

S503, the Hive data authentication agent sends the HQL authority verification result, the rewritten HQL, and the table and field blood relationship analysis results to the JDBC-based HQL submission module.

In another aspect, the present invention also provides a Hive data permission control proxy layer system, which is used to implement the Hive data permission control proxy layer method according to the present invention.

Further, the system includes a group big data cluster management platform, which is used to activate and register tenants for various departments using the platform, and configure the permissions of each tenant to use the Hive database and HDFS file directory in advance through the Ranger Admin; it also includes Hive embedded in HiveServer2. Plugin and HDFS plugin for HDFS; used to periodically pull permission policies from Ranger Admin and save them in the local policy repository.

Further, it also includes a data authority management center, which is provided with a table and field authority module, a row filtering and field desensitization module, and a user authority management module; wherein, the table and field authority defines data authority from the metadata dimension ;Row filtering and field desensitization define data permissions from the data dimension; Hive data authentication agent unit performs HQL parsing, HQL rewriting, HQL permission verification, and HQL table and field bloodline analysis.

The beneficial effects of the present invention are as follows: 1) the technical solution architecture design of the present invention can meet the fine-grained data permission control requirements within the enterprise; 2) the technical solution implementation of the present invention is to realize HQL parsing, rewriting and permission verification based on the Hive native class , the accuracy and stability are guaranteed to a certain extent, and there is a certain degree of confidence in meeting the production and online requirements. 3) The Java classes DatablackHiveAuthorizerFactory and DatablackHiveAuthorizer provided by the present invention are both specific implementations of the Hive interface, which can realize data authentication at the employee/user level. 4 ) The installation and deployment of the present invention do not need to invade the group big data cluster management platform, and can meet the needs of a large part of actual production scenarios.

Description of drawings

1 shows a schematic diagram of the architecture design of the Hive data permission control proxy layer method and system according to an embodiment of the present invention;

2 shows a schematic diagram of a test table structure and its data according to an embodiment of the present invention;

3 shows a schematic diagram of a data permission policy table according to an embodiment of the present invention;

FIG. 4 shows a schematic diagram of a mapping relationship table between employees and data permission policies according to an embodiment of the present invention;

5 shows a schematic diagram of an HQL authentication process according to an embodiment of the present invention;

6 shows a schematic diagram of key configuration parameters of HQL security authentication, authentication, and blood relationship analysis according to an embodiment of the present invention;

(a) in FIG. 7 is a comparison diagram before and after HQL rewriting according to an embodiment of the present invention, and (b) is a schematic diagram of a query result;

8 shows a schematic diagram of an HQL authentication process according to an embodiment of the present invention;

FIG. 9 shows a schematic diagram of a table and field blood relationship analysis process of HQL according to an embodiment of the present invention.

Detailed ways

The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are a part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc. The indicated orientation or positional relationship is based on the orientation or positional relationship shown in the accompanying drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the indicated device or element must have a specific orientation or a specific orientation. construction and operation, and therefore should not be construed as limiting the invention. Furthermore, the terms "first", "second", and "third" are used for descriptive purposes only and should not be construed to indicate or imply relative importance.

In the description of the present invention, it should be noted that the terms "installed", "connected" and "connected" should be understood in a broad sense, unless otherwise expressly specified and limited, for example, it may be a fixed connection or a detachable connection Connection, or integral connection; can be mechanical connection, can also be electrical connection; can be directly connected, can also be indirectly connected through an intermediate medium, can be internal communication between two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific situations.

Specific embodiments of the present invention will be described in detail below with reference to FIGS. 1 to 9 . It should be understood that the specific embodiments described herein are only used to illustrate and explain the present invention, but not to limit the present invention.

Definitions of terms involved in the text:

Hadoop: In a narrow sense, it refers to an open source distributed computing platform developed by the Apache Foundation, and in a broad sense, it refers to the ecology of big data components with "Hadoop" as the core;

HDFS: a distributed file storage system in the "Hadoop" ecosystem;

HBase: a non-relational database system based on columnar storage in the "Hadoop" ecosystem;

Hive: An offline query engine in the "Hadoop" ecosystem, providing a SQL dialect called "HiveQL (HQL)" to store, query and analyze large-scale data stored in Hadoop;

Spark: a memory-based distributed computing framework and offline computing engine in the "Hadoop" ecosystem;

Flink: A real-time computing engine that can realize stream-batch integration in the "Hadoop" ecosystem;

MapReduce: One of the most basic distributed computing frameworks and programming models in the "Hadoop" ecosystem;

Yarn: a general resource management system in the "Hadoop" ecosystem;

CDH: Hadoop stable release and big data cluster management platform;

HDP: open source big data cluster management platform;

EMR: Hadoop stable release and big data cluster management platform;

Ranger: A data permission management and control framework and components based on permission policy in the "Hadoop" ecosystem, which has been integrated by HDP and EMR;

Sentry: a role-based data permission control framework and components in the "Hadoop" ecosystem, which has been integrated by CDH;

Ranger Admin: Ranger's permission policy and component management center;

HiveServer2: A service provided by Hive that enables clients to execute HQL;

Kerberos: A computer network authorization protocol supported by the "Hadoop" ecosystem, used to authenticate personal communications in a secure manner in a non-secure network;

Keytab: The authentication ticket issued by the Kerberos server to the tenant;

JDBC: An application program interface used in the Java language to standardize client programs accessing databases, generally referring to Java database connections;

HiveConf: Java class provided in Hive source code, used to store Hadoop and Hive configuration in memory, inherited from Hadoop configuration class Configuration;

SessionState: The Java class provided in the Hive source code to create a Hive session and maintain the state in the session;

QueryState: Java class provided in the Hive source code to maintain the state information of an HQL query, such as the operation type corresponding to HiveConf and HQL;

ParseDriver: The Java class provided in the Hive source code, the provided parse method can parse an HQL into the corresponding abstract syntax tree node ASTNode;

ASTNode: Java class provided in Hive source code, used to represent an abstract syntax tree of HQL;

SemanticAnalyzerFactory: The Java class provided in the Hive source code, the provided get method can generate the corresponding SemanticAnalyzer according to ASTNode and QueryState;

SemanticAnalyzer: The Java class provided in the Hive source code. The analyze method provided can perform semantic analysis and optimization on ASTNode, parse metadata information such as tables and fields involved, and perform row filtering and field desensitization rewriting for HQL;

Context: The Java class provided in the Hive source code, which provides the context for SemanticAnalyzer and maintains the Token stream in HQL. After the SemanticAnalyzer rewrites, the HQL Token stream is stored in the Context;

TableMask: The Java class provided in the Hive source code rewrites the HQL through the row filtering and field desensitization expressions obtained from the HiveAuthorizer, and saves the rewritten HQL in the Context. The TableMask object is generated when the SemanticAnalyzer object is generated. generate;

HiveAuthorizer: The Java interface provided in the Hive source code defines a series of unimplemented methods, among which the checkPrivileges method is used for user data permission verification, the needTransform method is used to mark whether HQL is rewritten, and the applyRowFilterAndColumnMasking method is used to obtain table row filtering and field removal. sensitive expression;

DatablackHiveAuthorizer: An implementation class of the HiveAuthorizer interface class in the Hive source code, providing specific implementations of three methods: checkPrivileges, needTransform and applyRowFilterAndColumnMasking;

HiveAuthorizerFactory: The Java interface provided in the Hive source code defines an unimplemented createHiveAuthorizer method, which is the factory class of HiveAuthorizer;

DatablackHiveAuthorizerFactory: An implementation class of the HiveAuthorizerFactory interface class in the Hive source code, which implements the createHiveAuthorizer method to generate an instance object of DatablackHiveAuthorizer;

Driver: The Java class provided in the Hive source code is the driver of Hive, which is used to parse, compile, optimize and execute HQL, and provide a static doAuthorization method to verify user data permissions;

HiveOperationType: The Java enumeration class provided in the Hive source code is used to represent the operation type of HQL, such as adding a database, table query, table insertion, etc.;

HivePrivilegeObject: A Java class provided in the Hive source code to represent an object to be authenticated, which records information such as database, table, field, partition, and operation type;

QueryPlan: Java class provided in Hive source code, used to record the input and output format and query plan corresponding to an HQL;

HookContext: The Java class provided in the Hive source code is used to provide context for the hook class to execute the run method before and after HQL is executed, such as QueryPlan and QueryState;

LineageLogger: The Java class provided in the Hive source code is a HQL post-execution hook class, which is used to parse the lineage of HQL tables and fields, and then print them as logs;

ColumnLineageAnalysis: The inheritance class of the LineageLogger class in the Hive source code, rewrites the run method of the LineageLogger class, and can output the blood relationship of HQL tables and fields in a specific format.

FIG. 1 is an overall technical solution architecture design diagram of the fine-grained Hive data permission control proxy layer method and system of the present invention. As shown in the lower part of Figure 1, in the system, the security center is responsible for managing and maintaining the group big data cluster management platform 100, opening and registering tenants for each department using the platform, and configuring each tenant to use the Ranger Admin 120 in advance Hive database and HDFS file directory permissions. The security center is located in the big data and security department 110 shown in FIG. 1 . The Hive plug-in 131 embedded in the HiveServer2 130 and the HDFS plug-in 141 of the HDFS 140 periodically pull the permission policy from the Ranger Admin 120 and save it in the local policy repository. When an upper-level tenant submits HQL to HiveServer2130, it will go through four stages of HQL parsing, compilation, optimization, and execution. In the HQL compilation stage, the HiveServer2 130 can trigger the Hive plug-in 131 to authenticate the input/output privilege object (Hive Privilege Object) parsed by the HQL. The Hive plug-in 131 checks the permission objects with the locally cached permission policies one by one. If a permission object is identified as not allowed, it immediately interrupts the HQL execution of HiveServer2 130, and sends the authentication audit log to the Ranger Admin 120, and returns to the tenant Authentication failure details. Only after the Hive plug-in 131 is authenticated, the HiveServer2 130 submits a MapReduce task to the Yarn 150 to execute HQL. Since Hive's database and tables are stored in HDFS 140, HDFS plug-in 141 performs data authentication for tenants who bypass HiveServer2 130 and directly access HDFS 140. Here, when the Hive data permission policy is created, the corresponding HDFS 140 data permission policy will be created synchronously.

As shown in the middle part of Figure 1, the data analysis platform 200 of department A and the data analysis platform 300 of department B rely on tenant A.keytab and tenant B.keytab, respectively, after passing kerberos authentication, and the big data components on the group big data cluster management platform 100 interact. It should be noted that although the Ranger Admin120 on the Group's big data cluster management platform 100 can manage tenant resource permissions and realize resource isolation between tenants, data isolation between tenants is a coarse-grained data permission control scheme that cannot be used. Meet the more fine-grained data permission requirements of department A and department B for internal employees. Therefore, the innovation and contribution of the present invention mainly focus on the technical implementation of the data rights management center 400 and the Hive data authentication agent module 500 in the upper part of FIG. 1 .

According to the improved technical principle of the present invention, in the data authority management center 400, the Hive data authority categories are divided into table and field authority, row filtering and field desensitization, and the data authority management center 400 is correspondingly provided with tables and fields Permissions module 410 and row filtering and field desensitization module 420. Among them, table and field permissions define data permissions from the metadata dimension, such as table deletion, modification, clearing, and field query permissions; row filtering and field desensitization define data permissions from the data dimension. The user rights management module 430 is responsible for maintaining the mapping relationship between department employees and data rights policies. Specifically, within department A 200, employee A.b 221 applies for Hive data permission to department A 200 through the data permission approval service 210. After the data permission is approved, the data permission management center 400 synchronously creates a corresponding data permission policy, which is stored in the table and field permission module 410 and row filtering and field desensitization module 420, and update the mapping relationship between employees and data permission policies in user permission management 430.

The Hive data authentication agent unit 500 is the core of the data authorization agent layer scheme of the present invention, and is responsible for performing HQL rewriting 510, HQL authorization verification 520, and providing HQL table and field lineage analysis services 530. As shown in the upper part of Figure 1, employee A.b 221 of department A 200 submits a Hive task after applying for Hive table permission, and the JDBC-based HQL submission module 230 will send HQL and employee A.b's identity information to Hive data together The authentication proxy unit 500 performs authentication. First, the Hive data authentication proxy unit 500 pulls the user rights policy from the data rights management center 400 . Then, if there is a row filtering and field desensitization permission policy for employee A.b, the Hive data authentication agent 500 will rewrite the HQL during HQL parsing, such as adding row filtering and field desensitization expressions, so as to realize row filtering and field desensitization sensitive. Similar to the Hive plug-in of Ranger, after parsing the input/output permission objects of HQL, the Hive data authentication proxy unit 500 checks the permission objects and the locally cached permission policies one by one, if a certain permission object is identified as not allowed , then immediately notify the JDBC-based HQL submission module 230 to suspend HQL task submission and user authentication failure details, and send the authentication audit log to the data rights management center 400 . Finally, after the authentication is passed, the Hive data authentication proxy unit 500 performs table and field bloodline analysis on HQL, and sends the authentication result, the rewritten HQL, and table and field bloodline information to the JDBC-based HQL submission module 230. The JDBC-based HQL submission module 230 submits the rewritten HQL to the HiveServer2 130 on the big data management platform 100 for execution.

As shown in FIG. 1 , the data rights management center 400 and the Hive data authentication agent unit 500 are both outside the group big data cluster management platform 100 , and their installation and deployment do not need to invade the group big data cluster management platform 100 . In addition, Hive data authentication is performed before HQL is submitted to the big data cluster management platform 100, which does not interfere with the execution process of HiveServer2 130 and avoids the occurrence of security problems. The most important thing is that the granularity of Hive data authentication can be refined to employees within the department to meet the needs of finer-grained data permission control within the department. Therefore, the data rights management center 400 and the Hive data authentication proxy unit 500 are the data rights control proxy layers of the present invention.

The technical solutions of the fine-grained Hive data authority control proxy layer method and system of the present invention are as follows, and the method includes the following steps:

Step S1: Hive data permission application.

As shown in Fig. 1, when the department's data permission approval service approves the employee's Hive data permission application, the data permission management center 400 creates the corresponding data permission policy synchronously, and stores it in the table and field permission module 410 and row filtering and field masking In module 420, the mapping relationship between employees and data permission policies in user rights management 430 is updated.

In a specific embodiment, the Hive database applied by the department A 200 in the big data cluster management platform 100 is "hive_test", which has a Hive table named "ods_tbl_test", and its table structure and data are shown in FIG. 2 . Employee A.b 221 applies to the data permission approval service 210 for the viewing permission of the table "ods_tbl_test", and sets row filtering and field masking conditions. After the approval is passed, the data permission policy stored in the data permission management center 400 is shown in Figure 3, in which there are three permission policies, namely 1) full field query permission for the table "ods_tbl_test" of the database "hive_test", 2) permission for the database Row filtering of table "ods_tbl_test" of "hive_test", and 3) desensitization of fields of table "ods_tbl_test" of library "hive_test". Correspondingly, the mapping relationship between employees and data permission policies is shown in Figure 4. Employee A.a 222 has not applied for any permission yet, while the permission policy ID set of employee A.b 221 is {1, 2, 3}, which corresponds to the three permission policies in FIG. 3 . When an employee submits a Hive task, the JDBC-based HQL submission module 230 will send the HQL together with the employee's identity information to the Hive data authentication agent unit 500 for authentication. The process of HQL authentication is divided into the following four steps: HQL parsing, rewriting, authority checksum table, and field blood relationship analysis. The corresponding technical implementation process is shown in Figure 5. HiveConf, SessionState, QueryState, ParseDriver, ASTNode, SemanticAnalyzer, Driver, QueryPlan, and HookContext in the technical implementation flowchart are all Java classes provided by Hive. The following steps S2-S5 will describe the HQL authentication process in FIG. 5 in detail.

Step S2: HQL parsing.

Before HQL analysis, it is necessary to use the tenant keytab (Note: the keytab file is the identity authentication ticket issued by the Kerberos server to the tenant) that the department interacts with the group's big data cluster management platform 100 for kerberos authentication, so as to achieve data isolation between tenants. After the authentication is passed, create a HiveConf. The creation of HiveConf relies on the Hadoop and Hive configuration files provided by the big data cluster management platform 100. In addition, some key parameters must be set in the process of HiveConf creation to realize HQL row filtering, field desensitization and data authentication. These key parameters are described in detail in FIG. 6 , wherein the HQL security authentication manager is a Java class provided by the present invention, and its main function is to perform authority policy pulling and HQL data authentication, which is also the technical core of the present invention.

HQL parsing includes the following sub-steps:

S201, use HiveConf to create a SessionState object, and set the userName of the SessionState object to the employee account submitting the HQL.

S202, start the SessionState object, set the current database to be the Hive database applied for by the department on the big data cluster management platform, such as "hive_test" in the example of step S1, and initialize the transaction manager. After the SessionState object is created and started, it will be valid and unique in the global scope. It can communicate with Hadoop to submit distributed tasks, and it can also connect to Hive's metadata database to query metadata information.

S203, QueryState, Context, and ParseDriver objects are created in sequence. Call the parse method of the ParseDriver object to parse the original HQL into an abstract syntax tree node (Abstract Syntax Tree Node, ASTNode). Use the get method of Hive's SemanticAnalyzerFactory to generate the SemanticAnalyzer corresponding to QueryState and ASTNode.

S204, call the analyze method of SemanticAnalyzer to analyze the HQL.

Step S3: HQL rewriting.

In the process of parsing HQL, SemanticAnalyzer will perform row filtering and field desensitization rewriting of HQL through Hive's TableMask object. The detailed process of HQL rewriting includes the following sub-steps:

S301, traverse and parse the ASTNode of HQL to obtain table and field information;

S302, pull the row filtering and field desensitization permission policies of the table and field corresponding to the userName of SessionState from the data authority management center through the DatablackHiveAuthorizer provided by the present invention, and call the applyRowFilterAndColumnMasking method so that the TableMask object can correctly obtain the row filtering and field desensitization expressions Mode;

S303, rewrite the Token stream of the original HQL according to the row filtering and field desensitization expressions, and save the Token stream of the latest HQL in the Context object. For the example in step S1, assuming that the HQL submitted by employee A.b is "selectid, principal_part, note from ods_tbl_test", after pulling the permission policy in Figure 3, the HQL before and after TaskMask rewrite is shown in (a) in Figure 7 , the rewritten data query result is shown in (b) in Figure 7. By comparing the table data in Figure 2 and Figure 7 (b), it can be seen that the data has been row filtered and field desensitized according to the permission policy. Specifically, the original table ods_tbl_test in Figure 2 has 5 rows of records. The table and field data permission policy in the first row in Figure 3 grants employee A.b the full field query permission of the table ods_tbl_test. At the same time, the filtering strategy of the second row in Figure 3 makes the rewritten HQL in Figure 7 add a filter condition starting with where compared to the original HQL, and finally the query result of (b) in Figure 7 is only The last 2 rows of records in Figure 2, the first 3 rows of records are filtered during query because they do not meet the filtering conditions. In addition, the field desensitization strategy substr(note, 3) in line 3 in Figure 3 (Note: the data of the note field is cropped from the 3rd character) is applied to the note field of HQL after rewriting (a) in Figure 7, Make the note of the two records in (b) in Figure 7 a cropped version of the note in Figure 2 starting from the third character.

Step S4: HQL permission verification.

Based on QueryState, SemanticAnalyzer, and HQL, call the Driver's static doAuthorization method to implement HQL authorization verification. If the HQL permission verification succeeds, the method returns normally; otherwise, a permission verification failure exception is thrown. The bottom layer of the driver's authority verification relies on the DatablackHiveAuthorizer class provided by the present invention. The DatablackHiveAuthorizer class is the implementation class of Hive's HiveAuthorizer interface, which implements three methods: checkPrivileges, applyRowFilterAndColumnMasking, and needTransform. Figure 8 shows the underlying process details of Driver permission verification. Driver's doAuthorization will parse out HQL's HiveOperationType, input and output HivePrivilegeObject, and authentication context, and then call the checkPrivileges method of DatablackHiveAuthorizer. The checkPrivileges method will pull the user permission policy from the data permission management center, parse the tables, fields and operation types involved in the input and output HivePrivilegeObject, and then match the user permission policy. If all the input and output HivePrivilegeObject objects pass the permission check, the method returns normally, otherwise the permission check failure exception is thrown. Finally, the checkPrivileges method will also send the detailed log of the authentication audit to the data rights management center for audit analysis.

Step S5: HQL table and field blood relationship analysis.

After the HQL permission verification is passed, the blood relationship analysis of HQL tables and fields can be performed. Specifically include the following steps:

S501, based on HiveConf, QueryState, SemanticAnalyzer and HQL, create QueryPlan and HookContext objects.

S502, call the run method of the Java class ColumnLineageAnalysis provided by the present invention, and return the blood relationship of the table and the field in the HQL. ColumnLineageAnalysis is an inheritance class of Hive's LineageLogger. It rewrites the run method and can return HQL table and field blood relationship. For example, the blood relationship of the HQL table and field in (a) in Figure 7 is shown in Figure 9, that is, in the HQL query result in Figure 7, the target field id, principal_part and note come from the table in the Hive database hive_test respectively The fields id, principal_part and note of ods_tbl_test, where id and principal_part are the direct mapping from the source field to the target field, and note is transformed by the function of substr(note, 3).

S503, the Hive data authentication agent sends the HQL authentication result, the rewritten HQL, and the table and field blood relationship analysis results to the JDBC-based HQL submission module. If the HQL authentication is passed, the JDBC-based HQL submission module will submit the rewritten HQL to the group big data cluster management platform 100 for execution. Since the Hive data authentication proxy unit 500 performs data authentication for employees/users, it can meet the needs of more fine-grained data permission control within the enterprise.

The key technical points of the present invention include: 1) the technical solution architecture design of the present invention, such as the interaction logic and functional responsibilities between the data rights management center, the Hive data authentication agent and the JDBC-based HQL submission; The technical implementation scheme of HQL parsing, rewriting, authentication and table and field blood relationship analysis of Hive native classes; 3) The checkPrivileges method logic of the Java class DatablackHiveAuthorizer related to security authentication implemented by the present invention.

The HQL parsing and rewriting in step S3 can also be used to traverse the ASTNode through the visitor mode of ANTLR4, and analyze the tables, field resources and corresponding access methods involved in the HQL. However, the accuracy of this method is very limited, and it cannot cover complex HQL scenarios, so it usually cannot meet the production and launch standards.

In another aspect, the present invention also provides a fine-grained Hive data permission control proxy layer system, which is used to implement the fine-grained Hive data permission control proxy layer method according to the present invention.

In the description of this specification, description with reference to the terms "embodiment," "example," etc. means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the present invention . In this specification, schematic representations of the above terms are not necessarily directed to the same embodiment or example. Furthermore, those skilled in the art may combine or combine the different embodiments or examples described in this specification, as well as the features therein, without conflicting situations.

Although the foregoing content has shown and described the embodiments of the present invention, it should be understood that the foregoing embodiments are exemplary and should not be construed as limitations of the present invention. The above embodiments carry out update operations such as changes, modifications, substitutions, and modifications.

Claims (5)

1. A Hive data right control agent layer method, which is characterized in that the method comprises the following steps:

s1: hive data authority application; when the data authority approval service of a department approves the Hive data authority application of the employee, the data authority management center synchronously creates a corresponding data authority strategy, stores the corresponding data authority strategy in the table and field authority module and the row filtering and field desensitization module, and updates the mapping relation between the employee and the data authority strategy in the user authority management;

s2: resolving HQL; before HQL analysis, the Kerberos authentication of tenants needing to be carried out by utilizing Keytab of tenants interacting with a group big data cluster management platform by departments is needed, so that data isolation among the tenants is realized;

s3: HQL rewriting; in the process of analyzing the HQL, the SemanticAnalyzer performs line filtering and field desensitization rewriting on the HQL through a Hive TableMask object;

s4: checking HQL authority; based on QueryState, semanticAnalyzer and HQL, calling a static doAuthorization method of Driver to realize HQL data authority verification;

s5: HQL table, field blood margin analysis; when the HQL authority passes the verification, carrying out HQL table and field blood margin analysis;

after applying for the authority of the Hive table, the employee submits the Hive task, an HQL submitting module based on JDBC sends HQL and identity information of the employee to a Hive data authentication agent unit for authentication, and the HQL authentication process comprises HQL analysis, HQL rewriting, HQL authority verification, HQL table and field blood margin analysis;

in the step S2, after the kerberos authentication is passed, a HiveConf is created; the creation of HiveConf depends on Hadoop and Hive configuration files provided by a group big data cluster management platform, wherein the configuration files comprise core-site.xml, hdfs-site.xml, map-site.xml, yarn-site.xml and Hive-site.xml;

in step S2, HQL parsing includes the following sub-steps:

s201, creating a Session State object by using HiveConf, and setting the userName of the Session State object as an employee account submitted with HQL;

s202, starting a Session State object, setting a current database as a Hive database applied by a department on a big data cluster management platform, and initializing a transaction manager; after the sessionState object is created and started, the sessionState object is effective and unique, can communicate with Hadoop to submit distributed tasks, and can also be connected with a Hive metadata base to inquire metadata information;

s203, sequentially creating QueryState, context and ParseDriver objects; calling a parse method of the ParseDriver object to analyze the original HQL into abstract syntax tree nodes; generating a SemanticAnalyzer corresponding to QueryState and ASTNode by using a get method of Hive SemanticAnalyzer factor;

s204, calling an analyze method of the SemanticAnalyzer to analyze the HQL;

in step S3, the HQL rewriting process includes the following substeps:

s301, traversing and analyzing ASTNode of HQL to obtain table and field information;

s302, a table corresponding to the userName of the Sessionstate and a row filtering and field desensitization authority strategy of a field are pulled from a data authority management center through a DatablackHiveAuthorizer, and an applyRowFilterAndcolumnaging method is called so that a TableMask object can correctly acquire a row filtering and field desensitization expression;

s303, rewriting the Token stream of the original HQL according to the line filtering and field desensitization expression, and storing the latest Token stream of the HQL in a Context object;

in the step S4, if the HQL authority verification is successful, returning normally, otherwise, throwing out the abnormal authority verification failure; wherein, the authority verification bottom layer of the HiveDriver depends on a DatabeackHiveAuthorizer class; the DataBlackHiveAuthorizer class is an implementation class of a HiveAuthorizer interface of Hive, and realizes methods of checkPrivileges, applyRowFilterAndColumnMasking and needTransform; the doAuthorization of the Driver analyzes the HiveOperationType, the input and output HivePrivilegObjects and the authentication context of the HQL and then calls a checkPrivileges method of a DatablackHiveAuthorizer; the checkPrivileges method pulls a user authority policy to the data authority management center, analyzes tables and fields related in the input and output HiveprivileObject and operation types, and then matches the user authority policy; if all the input and output HivePrivilegObject objects pass the permission verification, the method returns normally, otherwise, the permission verification failure exception is thrown out;

in step S5, the HQL table and field blood relationship analysis specifically includes the following steps:

s501, based on HiveConf, queryState, semanticAnalyzer and HQL, creating QueryPlan and HookContext objects;

s502, calling a run method of Java ColumnLinearanalysis, and returning the table and field blood relationship in the HQL; columnLineageAnalysis is an inheritance class of the LineageLogger of Hive, a run method in the inheritance class is rewritten, and the table and field blood relationship of HQL can be returned;

and S503, the Hive data authentication agent sends the HQL authority verification result, the rewritten HQL, the table and field blood margin analysis result to an HQL submission module based on JDBC.

2. The Hive data authority control agent layer method of claim 1, wherein if HQL authentication passes, the JDBC-based HQL submission module submits the rewritten HQL to a group big data cluster management platform for execution.

3. A Hive data authority control proxy layer system, characterized in that the system is used for realizing the Hive data authority control proxy layer method according to any one of claims 1-2.

4. The Hive data authority control agent layer system according to claim 3, wherein the system comprises a group big data cluster management platform, is used for opening and registering tenants for each department using the platform, and configures the Hive database and HDFS file directory authority for each tenant in advance through Range Admin; the HDFS plug-in also comprises a Hive plug-in and an HDFS plug-in which are embedded into the HiveServer 2; the system is used for periodically pulling the authority policy from the Ranger Admin and storing the authority policy in a local policy repository.

5. The Hive data right control agent layer system according to claim 3 or 4, further comprising a data right management center, wherein the data right management center is provided with a table and field right module, a row filtering and field desensitizing module and a user right management module; wherein, the table and field authority defines data authority from metadata dimension; line filtering and field desensitization define data permissions from data dimensions;

and the Hive data authentication agent unit performs HQL analysis, HQL rewriting, HQL authority verification and HQL table and field blood margin analysis services.

Images