CN117675330A - Data platform access control method and device, storage medium and electronic equipment - Google Patents
Data platform access control method and device, storage medium and electronic equipment Download PDFInfo
- Publication number
- CN117675330A CN117675330A CN202311635565.2A CN202311635565A CN117675330A CN 117675330 A CN117675330 A CN 117675330A CN 202311635565 A CN202311635565 A CN 202311635565A CN 117675330 A CN117675330 A CN 117675330A
- Authority
- CN
- China
- Prior art keywords
- component
- data platform
- hive table
- atlas
- hive
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 59
- 238000003860 storage Methods 0.000 title claims abstract description 17
- 230000006870 function Effects 0.000 claims description 24
- 238000012795 verification Methods 0.000 claims description 10
- 238000013507 mapping Methods 0.000 claims description 9
- 230000008569 process Effects 0.000 claims description 9
- 230000007246 mechanism Effects 0.000 claims description 6
- 238000004891 communication Methods 0.000 claims description 4
- 238000013523 data management Methods 0.000 abstract description 5
- 238000007726 management method Methods 0.000 description 16
- 238000010586 diagram Methods 0.000 description 7
- 238000004590 computer program Methods 0.000 description 6
- 230000008676 import Effects 0.000 description 4
- 238000012423 maintenance Methods 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 230000008901 benefit Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 2
- 238000012550 audit Methods 0.000 description 2
- 238000013475 authorization Methods 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 238000010276 construction Methods 0.000 description 2
- 238000001914 filtration Methods 0.000 description 2
- 238000002372 labelling Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 230000035945 sensitivity Effects 0.000 description 2
- 239000008280 blood Substances 0.000 description 1
- 210000004369 blood Anatomy 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000000586 desensitisation Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000007619 statistical method Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The data platform access control method, the device, the storage medium and the electronic equipment are applied to public financial big data platforms, and the public financial big data platforms comprise a Knox gateway component, a Ranger component and an Atlas component. The public financial big data platform is provided with the external access address by using the Knox gateway component, the exposure of internal cluster architecture information can be reduced, the data is classified in a multi-dimensional mode by combining the Atlas component and the Ranger component by using the classification tag, the refined data access control function of the public financial big data platform is realized, and the comprehensive data management capability of the public financial big data platform is remarkably improved.
Description
Technical Field
The disclosure relates to the technical field of big data, and in particular relates to a data platform access control method, a data platform access control device, a storage medium and electronic equipment.
Background
As banks continue to build deep in the digitizing field, commercial banks have accumulated vast amounts of data. In the public service field, in order to improve public service quality and customer satisfaction, the data value of public financial data needs to be mined by means of a data platform.
However, due to the characteristics of complex structure, huge data volume and information sensitivity to public and financial data, the data platform is required to provide efficient integrated management capability of data.
Therefore, how to improve the data comprehensive management capability of the data platform becomes a technical problem that needs to be solved by those skilled in the art.
Disclosure of Invention
In view of the above problems, the present disclosure provides a data platform access control method, apparatus, storage medium and electronic device, which overcome or at least partially solve the above problems, and the technical solutions are as follows:
the utility model provides a data platform access control method, is applied to public finance big data platform, public finance big data platform includes Knox gateway component, range subassembly and Atlas subassembly, the method includes:
obtaining a source file of public financial data;
loading the public financial data source file into a pre-constructed Hive table;
importing target metadata in the Hive table into the Atlas component to obtain a classification tag added to the target metadata;
synchronizing the classification tag to the Ranger component to obtain a user permission configuration strategy corresponding to the classification tag;
associating the Hive table in the Spring background program of the public financial big data platform so as to carry out authority verification on the access control operation of the Hive table according to the user authority configuration strategy;
and using the Knox gateway component to map service running addresses of the Atlas component, the range component and the Spring background program to external access addresses respectively, so that a user initiates access control operation to the Hive table through the external access addresses.
Optionally, before the loading the pair of public financial data source files into the pre-constructed Hive table, the method further includes:
the Hive table is constructed using HiveQL statements.
Optionally, after the importing the target metadata in the Hive table into the Atlas component, the method further includes:
and configuring a Hive Hook function for the Hive table, so as to inform the Atlas component of the updated Hive table through the Hive Hook function.
Optionally, after configuring the Hive Hook function for the Hive table, the method further includes:
and constructing a Kafka queue between the Atlas component and the range component to push the updated metadata in the Atlas component to the range component through the Kafka queue.
Optionally, the synchronizing the class label to the range component, to obtain a user permission configuration policy corresponding to the class label, includes:
starting the Ranger component, and synchronizing the classification labels from the Atlas component by using a Ranger Tagsync process in the Ranger component;
and creating a user authority configuration strategy corresponding to the classification label by using the management page of the Ranger component.
Optionally, the user authority configuration policy includes operation authority configuration information and data security configuration information which specify that the user group has to the metadata under the classification label.
Optionally, the associating the Hive table in the Spring background program of the public financial big data platform includes:
and in the Spring background program of the public financial big data platform, the Hive table is associated by using a Java database connection mechanism.
A data platform access control device for a public financial big data platform, the public financial big data platform comprising a Knox gateway component, a range component and an Atlas component, the device comprising: a data source file obtaining unit, a data source file loading unit, a classification label adding unit, a user authority configuration strategy obtaining unit, a table association unit and an address mapping unit,
the data source file obtaining unit is used for obtaining a public financial data source file;
the data source file loading unit is used for loading the pair of public financial data source files into a pre-constructed Hive table;
the classification tag adding unit is used for importing target metadata in the Hive table into the Atlas component to obtain a classification tag added to the target metadata;
the user authority configuration strategy obtaining unit is used for synchronizing the classification labels to the Ranger component to obtain user authority configuration strategies corresponding to the classification labels;
the table association unit is used for associating the Hive table in the Spring background program of the public financial big data platform so as to carry out authority verification on the access control operation of the Hive table according to the user authority configuration strategy;
the address mapping unit is configured to map service running addresses of the Atlas component, the range component, and the Spring daemon to external access addresses respectively by using the Knox gateway component, so that a user initiates an access control operation to the Hive table through the external access addresses.
A computer-readable storage medium having a program stored thereon, wherein the program, when executed by a processor, implements the data platform access control method of any one of the above.
An electronic device comprising at least one processor, and at least one memory, bus connected to the processor; the processor and the memory complete communication with each other through the bus; the processor is configured to invoke the program instructions in the memory to perform the data platform access control method of any of the above.
By means of the technical scheme, the data platform access control method, the data platform access control device, the storage medium and the electronic equipment are applied to public financial big data platforms, and the public financial big data platforms comprise a Knox gateway component, a Ranger component and an Atlas component. The present disclosure may obtain a source file for public financial data; loading a public financial data source file into a pre-constructed Hive table; importing target metadata in the Hive table into an Atlas component to obtain a classification tag added to the target metadata; synchronizing the classification labels to a Ranger component to obtain a user permission configuration strategy corresponding to the classification labels; associating the Hive table in a Spring background program of the public financial big data platform so as to carry out authority verification on the access control operation of the Hive table according to a user authority configuration strategy; and using a Knox gateway component to map service running addresses of the Atlas component, the range component and the Spring background program to external access addresses respectively, so that a user initiates access control operation to the Hive table through the external access addresses. The public financial big data platform is provided with the external access address by using the Knox gateway component, the exposure of internal cluster architecture information can be reduced, the data is classified in a multi-dimensional mode by combining the Atlas component and the Ranger component by using the classification tag, the refined data access control function of the public financial big data platform is realized, and the comprehensive data management capability of the public financial big data platform is remarkably improved.
The foregoing description is merely an overview of the technical solutions of the present disclosure, and may be implemented according to the content of the specification in order to make the technical means of the present disclosure more clearly understood, and in order to make the above and other objects, features and advantages of the present disclosure more clearly understood, the following specific embodiments of the present disclosure are specifically described.
Drawings
Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the disclosure. Also, like reference numerals are used to designate like parts throughout the figures. In the drawings:
FIG. 1 is a schematic flow chart of an embodiment of a method for controlling access to a data platform according to an embodiment of the disclosure;
FIG. 2 is a schematic flow chart of another embodiment of a method for controlling access to a data platform according to an embodiment of the disclosure;
FIG. 3 illustrates a system architecture diagram for a public financial big data platform provided by an embodiment of the present disclosure;
fig. 4 is a schematic structural diagram of a data platform access control device according to an embodiment of the present disclosure;
fig. 5 shows a schematic structural diagram of an electronic device provided by an embodiment of the disclosure.
Detailed Description
Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
In the public service field, in order to improve public service quality and customer satisfaction, the data value of public financial data needs to be mined by means of a data platform. However, due to the complex structure, huge data volume and information sensitive nature of public financial data, there is a need for higher requirements on security, compliance and data asset management capabilities of data platforms.
The common security control means of the existing data platform is mainly based on a Kerberos protocol of a Hadoop distributed system infrastructure, and the Kerberos protocol is used as a computer network authorization protocol, and can be used for carrying out identity authentication on personal communication in a non-secure network by using a secure means. However, the Kerberos protocol can only issue an access password to a user satisfying the authentication condition to control whether the user has authority to access the big data service, the user authority cannot be controlled with finer granularity, and an important component of the Kerberos protocol manages that the page URL address is directly exposed to a visitor outside the system, resulting in high security risk of the data platform from network attack. Meanwhile, the conventional data platform is difficult to effectively integrate public financial data in each database and each data table, so that data distribution of the public financial data is not centralized, and security risks such as data leakage, damage, tampering and loss are easy to occur.
The data platform access control method provided by the embodiment of the disclosure aims to provide a technology for controlling access to public financial data for bank users so as to reduce security risks such as data damage, leakage, tampering, loss and the like of a related bank IT system in software development and operation and maintenance testing.
The data platform access control method provided by the embodiment of the disclosure is applied to a public financial big data platform, and the public financial big data platform comprises a Knox gateway component, a range component and an Atlas component.
The Knox gateway component is also called Apache Knox, which is an application gateway for interacting with REST interfaces and user interfaces of one or more Hadoop clusters in a secure manner, supporting access to the clusters by accessing exposed REST interfaces and HTTP services without exposing node information within the clusters.
The Ranger component is a Hadoop cluster authority framework for providing complex data authorities for operation, monitoring and management, and provides a centralized management mechanism for managing all data authorities of the Hadoop ecological ring. The Ranger component supports fine-grained data access control over Hadoop ecology components. By operating the Ranger console, an administrator can easily control the user's access to databases, tables, and field rights through configuration policies. These policies can be set for different users and groups while permissions can be seamlessly interfaced with Hadoop. The main functions of the Ranger component include verification, authorization, audit, data encryption, security management and the like of the component.
The Atlas component is a metadata management and governance tool, can provide interfaces and a series of plug-ins, can conveniently import database metadata information into the Atlas component for analysis, also provides a management page Atlas Admin for managing metadata, and through the Atlas component, enterprises can establish asset catalogs for database metadata, classify and govern the assets, analyze the data and provide high-quality metadata information for data governance.
As shown in fig. 1, a flowchart of an implementation manner of a data platform access control method provided by an embodiment of the present disclosure may include:
s100, obtaining a public financial data source file.
Wherein the public financial data source file is a data file recorded with metadata related to a public financial transaction. The embodiment of the disclosure can acquire the public financial data source file from different databases and data tables in a banking system of a commercial bank.
S110, loading the public financial data source file into a pre-constructed Hive table.
Hive is a data warehouse tool based on Hadoop, and is used for extracting, converting and loading data, and can store, inquire and analyze large-scale data stored in Hadoop. The Hive table is a database table that supports the Hive data warehouse tool to map structured data files onto the Hive table.
Specifically, the embodiment of the disclosure can load the public financial data source file under the appointed file directory into a pre-constructed Hive table.
S120, importing the target metadata in the Hive table into an Atlas component to obtain a classification label added to the target metadata.
Specifically, after the Atlas component is started, the embodiment of the disclosure may import target metadata that needs to be managed by the Atlas component from the Hive table into the Atlas component, and add different classification tags (tags) to the target metadata by the Atlas component. For example: the existing Hive data table, company basic information table, has the following fields: the embodiment of the disclosure can add classification labels of 'company legal information' for three columns of company legal names, company legal contact mailboxes and company legal mobile phone numbers in Atlas components, and add classification labels of 'company operating conditions' for three columns of company registered capital, company liabilities and company annual profit. Note that one column may be added with a plurality of classification tags, for example: the "company registered capital" column may be added with a "company part" classification tag indicating the source of the data, or may be added with a "company management status" classification tag indicating the classification of the data. After the classification labels are added to the data in the Atlas component, the classification labels can be searched through the Atlas Admin management page, and information of all data columns added with the classification labels can be retrieved.
S130, synchronizing the classification labels to a Ranger component to obtain user permission configuration strategies corresponding to the classification labels.
Specifically, the embodiment of the disclosure can synchronize information of the class label from the Atlas component to the range component, and the user permission configuration policy is configured for the class label in the range component.
Optionally, the user rights configuration policy includes operation rights configuration information and data security configuration information specifying that the user group has for the metadata under the classification tag.
The operation authority configuration information comprises data access authority and data modification authority. The embodiment of the disclosure can set the operation authority of each user group on the metadata under any classification label in the user authority configuration strategy of the classification label.
The data security configuration information comprises: 1. the use of desensitization operations limits the way users access to sensitive data. 2. Line level filtering is used to limit the way users query for certain specific data items. The embodiment of the disclosure can set the data security limiting mode of each user group to the metadata under the classification label in the user authority configuration strategy of any classification label.
And S140, associating the Hive table in a Spring background program of the public financial big data platform so as to carry out authority verification on the access control operation of the Hive table according to a user authority configuration strategy.
The public financial big data platform can be constructed based on a Spring framework. The Spring framework is an enterprise-level application framework of open source code. According to the embodiment of the disclosure, the Hive table is associated in the Spring background program provided for the public financial big data platform, so that the operation on each metadata in the Hive table can be executed only after conforming to the user permission configuration strategy configured in the range component.
S150, using a Knox gateway component to map service running addresses of the Atlas component, the range component and the Spring background program to external access addresses respectively, so that a user initiates access control operation to the Hive table through the external access addresses.
Wherein the service run address is a uniform resource locator (Uniform Resource Locator, URL).
After the Knox gateway component is started, service running addresses of the Atlas component, the range component and the Spring background program can be hidden in an address mapping mode, so that system architecture details of the public financial big data platform are shielded for external visitors, and the risk of network attack on the public financial big data platform is reduced.
Alternatively, the address mapping provided by the Knox gateway component may be as shown in table 1.
TABLE 1
The access control method for the data platform is applied to a public financial big data platform, and the public financial big data platform comprises a Knox gateway component, a Ranger component and an Atlas component. The present disclosure may obtain a source file for public financial data; loading a public financial data source file into a pre-constructed Hive table; importing target metadata in the Hive table into an Atlas component to obtain a classification tag added to the target metadata; synchronizing the classification labels to a Ranger component to obtain a user permission configuration strategy corresponding to the classification labels; associating the Hive table in a Spring background program of the public financial big data platform so as to carry out authority verification on the access control operation of the Hive table according to a user authority configuration strategy; and using a Knox gateway component to map service running addresses of the Atlas component, the range component and the Spring background program to external access addresses respectively, so that a user initiates access control operation to the Hive table through the external access addresses. The public financial big data platform is provided with the external access address by using the Knox gateway component, the exposure of internal cluster architecture information can be reduced, the data is classified in a multi-dimensional mode by combining the Atlas component and the Ranger component by using the classification tag, the refined data access control function of the public financial big data platform is realized, and the comprehensive data management capability of the public financial big data platform is remarkably improved.
Alternatively, embodiments of the present disclosure may construct Hive tables using HiveQL statements prior to step S110. Wherein the HiveQL statement is a Hive query language.
According to the embodiment of the disclosure, the Hive table is constructed through the hiveQL statement similar to the SQL statement, so that statistics of the MapReduce programming model can be realized quickly, the operation process of the MapReduce programming model is simpler, and the statistical analysis efficiency of the Hive table can be improved.
Optionally, the embodiment of the disclosure may configure a Hive Hook function on the Hive table after importing the target metadata in the Hive table into the Atlas component, so as to notify the Atlas component of the updated Hive table through the Hive Hook function. According to the embodiment of the disclosure, the Hive Hook function is configured on the Hive table, so that the change of metadata in the Hive table can be timely notified to the Atlas component.
Alternatively, embodiments of the present disclosure may construct a Kafka queue between Atlas components and range components after configuring a Hive Hook function for a Hive table, to push metadata updated in Atlas components to range components through the Kafka queue. According to the embodiment of the disclosure, updated metadata and classification labels in Atlas components can be pushed to a range component in real time through a Kafka queue to carry out user permission configuration.
Optionally, based on the method shown in fig. 1, as shown in fig. 2, a flowchart of another implementation of the data platform access control method provided in the embodiment of the present disclosure, step S130 may include:
s131, starting a range component, and synchronizing classification labels from the Atlas component by using a range Tagsync process in the range component.
S132, creating a user permission configuration strategy corresponding to the classification label by using a management page of the Ranger component.
Specifically, the embodiment of the disclosure may use the management page of the range component to create a user group, a user and a user authority configuration policy of the user group under each classification label in the Hive table. For example: the embodiment of the disclosure can create two user groups of a marketing person and a data analyzer and create users for banking staff below the user groups, and then can allocate the access right of the classification label of the company legal information to the user groups of the marketing person and the access right of the classification label of the company management condition to the user groups of the data analyzer. To prevent leakage of customer personal information, embodiments of the present disclosure may also desensitize corporate legal name fields, such as displaying only surnames and not displaying first names. In order to reduce the risk of disclosure to the greatest extent, the embodiment of the disclosure may employ line level filtering, and only open the client information access rights in the provincial and subsidiary business scope to the provincial and subsidiary data analyst.
Optionally, the embodiment of the disclosure may use a Java database connection mechanism to associate Hive tables in a Spring daemon on a public financial big data platform.
Among other things, the Java database connectivity mechanism (Java Database Connectivity, JDBC) is an application program interface in Java language that is used to specify how client programs access a database, providing methods such as querying and updating data in the database.
The system architecture for public financial big data platform provided by the embodiment of the disclosure can provide uniform access entry for URL by using Knox gateway component as shown in fig. 3, and provide uniform service request URL in format convenient for user to memorize while reducing exposure of internal cluster architecture information. The Atlas component is used for classifying and labeling the data according to the sensitivity degree, the data content, the data source and other dimensions, so that operation and maintenance analysts can conveniently clean complex relations such as blood edges, sources, flow directions, attributions and the like among mass data to meet the requirements of audit compliance, safety and traceability. By combining with the Ranger component, fine-grained data access control functionality may be provided to the data warehouse, such as authority management capabilities including read, write, create index, create lock, etc., at various levels of databases, tables, columns, rows, etc., may be provided to the Apache Hive data warehouse component (hereinafter "Hive").
The embodiment of the disclosure is based on the Knox gateway component, the Ranger component and the Atlas component to construct the system architecture of the public financial big data platform, and the Knox gateway component is used for shielding sensitive information such as technical architecture type selection and operation ports of the big data analysis platform, so that the possibility of the system being subjected to external security attack is reduced. The Atlas component is used for classifying and labeling data, and the range component is combined to provide a data access control function with fine granularity for a data warehouse (taking a Hive data table as an example), so that large data platform operation and maintenance personnel can intuitively classify and manage the data according to multiple dimensions such as different sources, access rights, purposes and the like, and the security risk caused by sensitive data leakage or tampering in the application development and production operation and maintenance process is reduced to the greatest extent.
Although operations are depicted in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order. In certain circumstances, multitasking and parallel processing may be advantageous.
It should be understood that the various steps recited in the method embodiments of the present disclosure may be performed in a different order and/or performed in parallel. Furthermore, method embodiments may include additional steps and/or omit performing the illustrated steps. The scope of the present disclosure is not limited in this respect.
Corresponding to the above method embodiment, the embodiment of the present disclosure further provides a data platform access control device, with a structure shown in fig. 4, which is applied to a public financial big data platform, and the public financial big data platform includes a Knox gateway component, a range component and an Atlas component, where the data platform access control device includes: a data source file obtaining unit 100, a data source file loading unit 200, a class label adding unit 300, a user authority configuration policy obtaining unit 400, a table associating unit 500, and an address mapping unit 600.
A data source file obtaining unit 100 for obtaining a source file of public financial data.
The data source file loading unit 200 is configured to load the public financial data source file into a Hive table that is built in advance.
The class label adding unit 300 is configured to import the target metadata in the Hive table into the Atlas component, and obtain a class label added to the target metadata.
The user right configuration policy obtaining unit 400 is configured to synchronize the class label to the range component, and obtain the user right configuration policy corresponding to the class label.
The table association unit 500 is configured to associate the Hive table in the Spring daemon of the public financial big data platform, so as to perform permission verification on the access control operation of the Hive table according to the user permission configuration policy.
The address mapping unit 600 is configured to map service running addresses of the Atlas component, the range component, and the Spring daemon to external access addresses respectively by using the Knox gateway component, so that a user initiates an access control operation to the Hive table through the external access addresses.
Optionally, the data platform access control device may further include: hive table building unit.
The Hive table construction unit is configured to construct a Hive table using HiveQL statements before the data source file loading unit 200 loads the public financial data source file into the Hive table constructed in advance.
Optionally, the data platform access control device may further include: hive Hook function configuration unit.
And the Hive Hook function configuration unit is configured to configure a Hive Hook function for the Hive table after the classification tag adding unit 300 imports the target metadata in the Hive table into the Atlas assembly, so as to notify the Atlas assembly of the updated Hive table through the Hive Hook function.
Optionally, the data platform access control device may further include: kafka queue building block.
And the Kafka queue construction unit is used for constructing a Kafka queue between the Atlas assembly and the range assembly after the Hive Hook function configuration unit configures the Hive Hook function for the Hive table so as to push the updated metadata in the Atlas assembly to the range assembly through the Kafka queue.
Optionally, the user permission configuration policy obtaining unit 400 is specifically configured to start a range component, and synchronize classification labels from Atlas components by using a range Tagsync process in the range component; a user permission configuration policy corresponding to the class label is created using a management page of the Ranger component.
Optionally, the user rights configuration policy includes operation rights configuration information and data security configuration information specifying that the user group has for the metadata under the classification tag.
Optionally, the table association unit 500 is specifically configured to associate Hive tables in a Spring daemon of the public financial big data platform by using a Java database connection mechanism.
The data platform access control device is applied to a public financial big data platform, and comprises a Knox gateway component, a Ranger component and an Atlas component. The present disclosure may obtain a source file for public financial data; loading a public financial data source file into a pre-constructed Hive table; importing target metadata in the Hive table into an Atlas component to obtain a classification tag added to the target metadata; synchronizing the classification labels to a Ranger component to obtain a user permission configuration strategy corresponding to the classification labels; associating the Hive table in a Spring background program of the public financial big data platform so as to carry out authority verification on the access control operation of the Hive table according to a user authority configuration strategy; and using a Knox gateway component to map service running addresses of the Atlas component, the range component and the Spring background program to external access addresses respectively, so that a user initiates access control operation to the Hive table through the external access addresses. The public financial big data platform is provided with the external access address by using the Knox gateway component, the exposure of internal cluster architecture information can be reduced, the data is classified in a multi-dimensional mode by combining the Atlas component and the Ranger component by using the classification tag, the refined data access control function of the public financial big data platform is realized, and the comprehensive data management capability of the public financial big data platform is remarkably improved.
The specific manner in which the individual units perform the operations in relation to the apparatus of the above embodiments has been described in detail in relation to the embodiments of the method and will not be described in detail here.
The data platform access control device comprises a processor and a memory, wherein the data source file obtaining unit 100, the data source file loading unit 200, the class label adding unit 300, the user authority configuration policy obtaining unit 400, the table association unit 500, the address mapping unit 600 and the like are all stored in the memory as program units, and the processor executes the program units stored in the memory to realize corresponding functions.
The processor includes a kernel, and the kernel fetches the corresponding program unit from the memory. The kernel can be provided with one or more than one, the Knox gateway component is used for providing an external access address for the public financial big data platform by adjusting kernel parameters, the information exposure of an internal cluster architecture is reduced, the Atlas component and the Ranger component are combined, the classification label is used for classifying data in a multi-dimensional manner, and the refined data access control function of the public financial big data platform is achieved, so that the comprehensive data management capability of the public financial big data platform is remarkably improved.
The disclosed embodiments provide a computer-readable storage medium having a program stored thereon, which when executed by a processor, implements the data platform access control method.
The embodiment of the disclosure provides a processor for running a program, wherein the program runs to execute the data platform access control method.
As shown in fig. 5, an embodiment of the present disclosure provides an electronic device 1000, the electronic device 1000 comprising at least one processor 1001, and at least one memory 1002, bus 1003 connected to the processor 1001; wherein, the processor 1001 and the memory 1002 complete communication with each other through the bus 1003; the processor 1001 is configured to call program instructions in the memory 1002 to perform the data platform access control method described above. The electronic device herein may be a server, a PC, a PAD, a mobile phone, etc.
The present disclosure also provides a computer program product adapted to perform a program initialized with the steps of the data platform access control method when executed on an electronic device.
The present disclosure is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus, electronic devices (systems), and computer program products according to embodiments of the disclosure. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In one typical configuration, the electronic device includes one or more processors (CPUs), memory, and a bus. The electronic device may also include input/output interfaces, network interfaces, and the like.
The memory may include volatile memory, random Access Memory (RAM), and/or nonvolatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM), among other forms in computer readable media, the memory including at least one memory chip. Memory is an example of a computer-readable medium.
Computer readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device. Computer-readable media, as defined herein, does not include transitory computer-readable media (transmission media), such as modulated data signals and carrier waves.
It should be noted that, the user information (including, but not limited to, user equipment information, user personal information, etc.) and the data (including, but not limited to, data for analysis, stored data, presented data, etc.) related to the present disclosure are information and data authorized by the user or sufficiently authorized by each party, and the collection, use, and processing of the related data need to comply with the related laws and regulations and standards of the related countries and regions.
In the description of the present disclosure, it should be understood that, if the directions or positional relationships indicated by the terms "upper", "lower", "front", "rear", "left" and "right", etc., are based on the directions or positional relationships shown in the drawings, are merely for convenience of describing the present invention and simplifying the description, and do not indicate or imply that the positions or elements referred to must have a specific orientation, be constructed and operated in a specific orientation, and thus should not be construed as limitations of the present disclosure.
It is noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article or apparatus that comprises an element.
It will be appreciated by those skilled in the art that embodiments of the present disclosure may be provided as a method, system, or computer program product. Accordingly, the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present disclosure may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.
The foregoing is merely exemplary of the present disclosure and is not intended to limit the present disclosure. Various modifications and variations of this disclosure will be apparent to those skilled in the art. Any modifications, equivalent substitutions, improvements, or the like, which are within the spirit and principles of the present disclosure, are intended to be included within the scope of the claims of the present disclosure.
Claims (10)
1. The utility model provides a data platform access control method, which is characterized in that the method is applied to public financial big data platform, the public financial big data platform includes Knox gateway component, ranger component and Atlas component, the method includes:
obtaining a source file of public financial data;
loading the public financial data source file into a pre-constructed Hive table;
importing target metadata in the Hive table into the Atlas component to obtain a classification tag added to the target metadata;
synchronizing the classification tag to the Ranger component to obtain a user permission configuration strategy corresponding to the classification tag;
associating the Hive table in the Spring background program of the public financial big data platform so as to carry out authority verification on the access control operation of the Hive table according to the user authority configuration strategy;
and using the Knox gateway component to map service running addresses of the Atlas component, the range component and the Spring background program to external access addresses respectively, so that a user initiates access control operation to the Hive table through the external access addresses.
2. The method of claim 1, wherein prior to said loading the pair of male financial data source files into a pre-built Hive table, the method further comprises:
the Hive table is constructed using HiveQL statements.
3. The method of claim 1, wherein after the importing of the target metadata in the Hive table into the Atlas component, the method further comprises:
and configuring a Hive Hook function for the Hive table, so as to inform the Atlas component of the updated Hive table through the Hive Hook function.
4. A method according to claim 3, wherein after configuring the Hive Hook function for the Hive table, the method further comprises:
and constructing a Kafka queue between the Atlas component and the range component to push the updated metadata in the Atlas component to the range component through the Kafka queue.
5. The method of claim 1, wherein synchronizing the class label to the range component obtains a user permission configuration policy corresponding to the class label, comprising:
starting the Ranger component, and synchronizing the classification labels from the Atlas component by using a Ranger Tagsync process in the Ranger component;
and creating a user authority configuration strategy corresponding to the classification label by using the management page of the Ranger component.
6. The method of claim 5, wherein the user rights configuration policy includes operating rights configuration information and data security configuration information specifying that a group of users have for the category label metadata.
7. The method of claim 1, wherein associating the Hive table in the Spring daemon of the public financial big data platform comprises:
and in the Spring background program of the public financial big data platform, the Hive table is associated by using a Java database connection mechanism.
8. A data platform access control device, characterized in that it is applied to a public financial big data platform, said public financial big data platform includes Knox gateway component, range component and Atlas component, said device includes: a data source file obtaining unit, a data source file loading unit, a classification label adding unit, a user authority configuration strategy obtaining unit, a table association unit and an address mapping unit,
the data source file obtaining unit is used for obtaining a public financial data source file;
the data source file loading unit is used for loading the pair of public financial data source files into a pre-constructed Hive table;
the classification tag adding unit is used for importing target metadata in the Hive table into the Atlas component to obtain a classification tag added to the target metadata;
the user authority configuration strategy obtaining unit is used for synchronizing the classification labels to the Ranger component to obtain user authority configuration strategies corresponding to the classification labels;
the table association unit is used for associating the Hive table in the Spring background program of the public financial big data platform so as to carry out authority verification on the access control operation of the Hive table according to the user authority configuration strategy;
the address mapping unit is configured to map service running addresses of the Atlas component, the range component, and the Spring daemon to external access addresses respectively by using the Knox gateway component, so that a user initiates an access control operation to the Hive table through the external access addresses.
9. A computer-readable storage medium having a program stored thereon, wherein the program, when executed by a processor, implements the data platform access control method according to any one of claims 1 to 7.
10. An electronic device comprising at least one processor, and at least one memory, bus connected to the processor; the processor and the memory complete communication with each other through the bus; the processor is configured to invoke program instructions in the memory to perform the data platform access control method of any of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311635565.2A CN117675330A (en) | 2023-11-30 | 2023-11-30 | Data platform access control method and device, storage medium and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311635565.2A CN117675330A (en) | 2023-11-30 | 2023-11-30 | Data platform access control method and device, storage medium and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117675330A true CN117675330A (en) | 2024-03-08 |
Family
ID=90063492
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311635565.2A Pending CN117675330A (en) | 2023-11-30 | 2023-11-30 | Data platform access control method and device, storage medium and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117675330A (en) |
-
2023
- 2023-11-30 CN CN202311635565.2A patent/CN117675330A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105357201B (en) | A kind of object cloud storage access control method and system | |
| US11256702B2 (en) | Systems and methods for management of multi-tenancy data analytics platforms | |
| US11227068B2 (en) | System and method for sensitive data retirement | |
| US20170091279A1 (en) | Architecture to facilitate organizational data sharing and consumption while maintaining data governance | |
| US9477574B2 (en) | Collection of intranet activity data | |
| US20100198804A1 (en) | Security management for data virtualization system | |
| US10445324B2 (en) | Systems and methods for tracking sensitive data in a big data environment | |
| US10360394B2 (en) | System and method for creating, tracking, and maintaining big data use cases | |
| US10503923B1 (en) | Centralized data store for multiple data processing environments | |
| US20090276446A1 (en) | Process and method for classifying structured data | |
| US20190050435A1 (en) | Object data association index system and methods for the construction and applications thereof | |
| CN113535846B (en) | Big data platform and construction method thereof | |
| CN113297433A (en) | Method and system for accessing graph database | |
| CN112905978A (en) | Authority management method and device | |
| CN112149112A (en) | Enterprise information security management method based on authority separation | |
| CN110928963A (en) | Column-level authority knowledge graph construction method for operation and maintenance service data table | |
| CN117675330A (en) | Data platform access control method and device, storage medium and electronic equipment | |
| KR101304452B1 (en) | A cloud system for document management using location | |
| WO2021034329A1 (en) | Data set signatures for data impact driven storage management | |
| Ospanova et al. | Building a model of the integrity of information resources within an enterprise management system | |
| Bellini et al. | Digital Identity: A Case Study of the ProCIDA Project | |
| US11886608B2 (en) | Subject logging | |
| Bao et al. | Large-Scale Scientific Research Instrument Resource Information Sharing Platform through O2O Mode | |
| Rouf et al. | Big Data in Smart Ecosystems: Trends, Challenges and Future Prospectus | |
| Parente et al. | Flexible Fine-grained Data Access Management for Hyperledger Fabric |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |