CN111737741B

CN111737741B - Distributed database cluster access method and intermediate service layer

Info

Publication number: CN111737741B
Application number: CN202010564789.9A
Authority: CN
Inventors: 刘雪晶; 林立成; 翁晓俊; 王之乐
Original assignee: Industrial and Commercial Bank of China Ltd ICBC
Current assignee: Industrial and Commercial Bank of China Ltd ICBC
Priority date: 2020-06-19
Filing date: 2020-06-19
Publication date: 2024-02-27
Anticipated expiration: 2040-06-19
Also published as: CN111737741A

Abstract

The embodiment of the application provides a distributed database cluster access method and an intermediate service layer, wherein the method comprises the following steps: according to cluster identifications of all target distributed database clusters, obtaining connection threads corresponding to all target distributed database clusters in a local search mode, creating all connection threads based on all configuration files for kerberos security authentication, acquiring all configuration files based on local merging authentication files respectively, wherein the merging authentication files are used for storing one-to-one correspondence between all cluster identifications and authentication configuration information; and accessing the target distributed database clusters by the target users based on the connection threads. According to the method and the device, the user can access to a plurality of distributed database clusters at the same time, so that the safety and the effectiveness of cross-cluster access of the user to the distributed database clusters can be effectively improved, and the effectiveness and the reliability of cross-cluster access of the user to the plurality of distributed database clusters can be improved.

Description

Distributed database cluster access method and intermediate service layer

Technical Field

The application relates to the technical field of data processing, in particular to a distributed database cluster access method and an intermediate service layer.

Background

With the rapid development of information technology, the exponential growth of structured and semi-structured data including characters, images, video and the like is realized. The conventional database is difficult to store and analyze the content of the data, so that the industry generally uses big data technologies such as the Hadoop ecosystem to perform business processing. The distributed database such as HBase is used as an important component of an ecological circle such as Hadoop, has the main characteristics of supporting real-time storage and inquiry of mass data, has the characteristics of high reliability, high performance, column orientation, scalability and the like, and is widely used in the industries such as electronic commerce, internet of things and the like.

As more and more enterprises or users use distributed databases, there is a significant problem inside each enterprise: each sub department or application has own application server, and the application servers are relatively independent, and under the cross-cluster scene, different cluster user data can not be shared, and the defect that the data can not be shared exists among independent systems.

At present, two common solutions exist, one is to use an unsafe authentication mode in a distributed database cluster of each service system, and cluster users interact with the cluster without verification, so that only configuration files of different clusters need to be loaded in sequence, different connection examples are created, and access to data of different clusters can be realized. However, this scheme has security problems and risks such as tampering with data by malicious users. Another approach may be to take the form of handling data, i.e., copying data across clusters. However, as the amount of service data increases further, more and more data is copied, this way significantly increases the running cost and causes data redundancy; meanwhile, when the clusters cannot communicate, the problem of mutual trust of the clusters is also required to be considered, so that cross-cluster data copying has certain limitation, and the difficulty and timeliness of data sharing are greatly increased. And in addition, the security verification is carried out on the cluster users, but in the authentication process, authentication files of a plurality of clusters are needed to be loaded in sequence, so that the clusters can cover configuration files, namely, after the authentication of the latter cluster is successful, the connection of the former cluster can be invalid, so that the requirement that the cluster users access the plurality of clusters at the same time can not be met, and the effectiveness of the distributed database clusters is reduced. That is, in any of the above access methods in the prior art, the security, timeliness, and effectiveness of the cross-cluster access of the user to the distributed database cluster cannot be satisfied at the same time.

Disclosure of Invention

Aiming at the problems in the prior art, the application provides a distributed database cluster access method and an intermediate service layer, which can enable a user to access a plurality of distributed database clusters at the same time, further can effectively improve the security and the effectiveness of cross-cluster access of the user to the distributed database clusters, and can improve the effectiveness and the reliability of cross-cluster access of the user to the plurality of distributed database clusters at the same time.

In order to solve the technical problems, the application provides the following technical scheme:

in a first aspect, the present application provides a distributed database cluster access method, including:

acquiring a distributed database cluster access request sent by a target user passing identity authentication, wherein the distributed database cluster access request comprises cluster identifiers of a plurality of target distributed database clusters;

if the target distributed database clusters are judged to be cross-user clusters which are authorized to be accessed by the target users, locally searching and obtaining connection threads corresponding to the target distributed database clusters according to cluster identifiers of the target distributed database clusters, wherein the connection threads are pre-created based on configuration files corresponding to the distributed database clusters and used for kerberos security authentication, the configuration files are pre-obtained based on a combined authentication file stored locally, and the combined authentication file is used for storing a one-to-one correspondence between the cluster identifiers of the distributed database clusters and authentication configuration information;

And accessing the target distributed database clusters by the target users based on the corresponding connection threads of the target distributed database clusters.

Further, before the obtaining the distributed database cluster access request sent by the target user passing the identity authentication, the method further comprises:

acquiring authentication configuration information corresponding to configuration files for performing kerberos security authentication, wherein the configuration files correspond to each distributed database cluster respectively;

storing the one-to-one correspondence between each authentication configuration information and cluster identifiers of each distributed database cluster into a combined authentication file;

acquiring configuration files corresponding to the distributed database clusters respectively based on the authentication configuration information in the combined authentication file;

storing the one-to-one correspondence between the configuration files and cluster identifications of the distributed database clusters to a local place;

establishing long connection with each distributed database cluster according to configuration files corresponding to each distributed database cluster, wherein each long connection corresponds to one connection thread;

and updating the authentication ticket TGT which is used for performing kerberos security authentication and corresponds to each distributed database cluster at regular time.

Further, before the obtaining the distributed database cluster access request sent by the target user passing the identity authentication, the method further includes:

generating a symmetric key by applying a symmetric encryption algorithm, and sending the symmetric key to a preset configuration center;

generating key objects of asymmetric private keys and public keys corresponding to each user by applying an asymmetric encryption algorithm, and encrypting each key object by applying the symmetric encryption algorithm;

storing each encrypted key object into a relational database, so that after receiving a registration request sent by each user, the configuration center obtains the encrypted key object corresponding to the target user from the relational database, and sends a public key plaintext, an encrypted symmetric key and an encrypted private key object corresponding to each key object to each corresponding user, so that each user decrypts the encrypted symmetric key based on the obtained public key plaintext, decrypts the private key object based on the corresponding obtained symmetric key, and correspondingly obtains a private key for identity authentication.

Further, the distributed database cluster access request also includes a user identifier of the target user;

correspondingly, before the cluster identification of each target distributed database cluster is searched locally to obtain the configuration file corresponding to each target distributed database cluster, the method further comprises the steps of:

searching authorized user identifiers corresponding to the cluster identifiers of the target distributed database clusters from an authority control table arranged in a relational database;

judging whether the authorized user identifications corresponding to the cluster identifications of the target distributed database clusters all contain the user identifications of the target users, if so, judging that the target distributed database clusters are cross-user clusters which are authorized to be accessed by the target users.

In a second aspect, the present application provides an intermediate service layer, comprising:

the request receiving module is used for acquiring a distributed database cluster access request sent by a target user passing through identity authentication, wherein the distributed database cluster access request comprises cluster identifiers of a plurality of target distributed database clusters;

the thread invoking module is used for locally searching and obtaining connection threads corresponding to the target distributed database clusters according to the cluster identification of the target distributed database clusters if the target distributed database clusters are judged to be cross-user clusters which are authorized to be accessed by the target users, wherein the connection threads are pre-created based on configuration files corresponding to the distributed database clusters and used for kerberos security authentication, the configuration files are pre-acquired based on a combined authentication file stored locally, and the combined authentication file is used for storing a one-to-one correspondence between the cluster identification of the distributed database clusters and authentication configuration information;

And the cluster access module is used for realizing the access of the target user to each target distributed database cluster based on the corresponding connection thread of each target distributed database cluster.

Further, the method further comprises the following steps: a thread creation module for performing the following:

Further, the method further comprises the following steps: the identity authentication module is used for executing the following contents:

Further, the distributed database cluster access request also comprises a user identifier of the target user;

Correspondingly, the intermediate service layer further comprises: and the permission query module is used for executing the following contents:

In a third aspect, the present application provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor implements the distributed database cluster access method when executing the program.

In a fourth aspect, the present application provides a computer readable storage medium having stored thereon a computer program which when executed by a processor implements the distributed database cluster access method.

According to the technical scheme, the distributed database cluster access method and the intermediate service layer provided by the application comprise the following steps: acquiring a distributed database cluster access request sent by a target user passing identity authentication, wherein the distributed database cluster access request comprises cluster identifiers of a plurality of target distributed database clusters; if the target distributed database clusters are judged to be cross-user clusters which are authorized to be accessed by the target users, locally searching and obtaining connection threads corresponding to the target distributed database clusters according to cluster identifiers of the target distributed database clusters, wherein the connection threads are pre-created based on configuration files corresponding to the distributed database clusters and used for kerberos security authentication, the configuration files are pre-obtained based on a combined authentication file stored locally, and the combined authentication file is used for storing a one-to-one correspondence between the cluster identifiers of the distributed database clusters and authentication configuration information; the method and the system can effectively improve the security and the effectiveness of cross-cluster access of the distributed database clusters by the user, can realize the access to a plurality of clusters at the same time, can ensure the effectiveness of the cross-cluster access of the distributed database clusters by the user, can effectively improve the reliability of the cross-cluster access of the distributed database clusters by the user, can effectively reduce the maintenance cost of the cross-cluster access, and can further improve the cluster access efficiency by storing the configuration file locally, thereby safely, quickly and effectively solving the problem that the cross-cluster data cannot be shared, improving the reliability and the intelligent degree of the operation of the distributed database clusters in enterprises, and effectively improving the user experience of the cluster access users.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, and it is obvious that the drawings in the following description are some embodiments of the present application, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

Fig. 1 is a flow chart of a distributed database cluster access method in an embodiment of the present application.

Fig. 2 is a schematic flowchart of steps 011 to 016 in the distributed database cluster accessing method in the embodiment of the present application.

Fig. 3 is a schematic flowchart of steps 021 to 023 in the distributed database cluster access method in the embodiment of the present application.

Fig. 4 is a schematic flow chart of steps 031 and 032 in the distributed database cluster access method in the embodiment of the present application.

Fig. 5 is a first structural schematic diagram of an intermediate service layer in an embodiment of the present application.

Fig. 6 is a second structural schematic diagram of an intermediate service layer in an embodiment of the present application.

Fig. 7 is a third structural schematic diagram of an intermediate service layer in an embodiment of the present application.

Fig. 8 is a fourth structural diagram of an intermediate service layer in an embodiment of the present application.

Fig. 9 is a schematic structural diagram of a distributed database cluster access system provided by an application example of the present application.

Fig. 10 is an initialization flow chart of the initialization module execution provided by the application example of the present application.

Fig. 11 is a flowchart of performing private key propagation of each tenant by the intermediate service layer tenant authentication module provided by the application instance of the present application.

Fig. 12 is a flowchart of a distributed database cluster access method provided by an application example of the present application.

Fig. 13 is a schematic diagram of an implementation process of the rights access rule provided by the application example of the present application.

Fig. 14 is a schematic flow chart of a creation and maintenance thread of the cluster connection creation and refresh module 4 provided by the application example of the present application.

Fig. 15 is a schematic structural diagram of an electronic device in an embodiment of the present application.

Detailed Description

For the purposes of making the objects, technical solutions and advantages of the embodiments of the present application more clear, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is apparent that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments, which can be made by one of ordinary skill in the art based on the embodiments herein without making any inventive effort, are intended to be within the scope of the present application.

In order to solve the problem that the existing distributed database cluster access mode cannot simultaneously meet the safety, timeliness and effectiveness of cross-cluster access of a user to a distributed database cluster, the application respectively provides an embodiment of a distributed database cluster access method, an intermediate service layer for realizing the distributed database cluster access method, electronic equipment and a computer readable storage medium, wherein the distributed database cluster access request sent by a target user passing identity authentication is obtained, and the distributed database cluster access request comprises cluster identifiers of a plurality of target distributed database clusters; if the target distributed database clusters are judged to be cross-user clusters which are authorized to be accessed by the target users, locally searching and obtaining connection threads corresponding to the target distributed database clusters according to cluster identifiers of the target distributed database clusters, wherein the connection threads are pre-created based on configuration files corresponding to the distributed database clusters and used for kerberos security authentication, the configuration files are pre-obtained based on a combined authentication file stored locally, and the combined authentication file is used for storing a one-to-one correspondence between the cluster identifiers of the distributed database clusters and authentication configuration information; based on the corresponding connection thread of each target distributed database cluster, the target user can access each target distributed database cluster, a kerberos security authentication mechanism is introduced, the problem of cluster security authentication is solved, an intermediate service layer is packaged, the problem of authentication failure of single-process access to a plurality of distributed database clusters is overcome in the intermediate service layer, a multi-tenant authentication and authentication mechanism is provided, the connection of access to a plurality of clusters is maintained in the intermediate service layer, so that users of different distributed database clusters can safely inquire and access tables and data in other clusters across clusters, and the problem that inter-cluster data cannot be accessed mutually is solved at low cost.

The following examples are given by way of illustration.

In one or more embodiments herein, an example of a distributed database cluster may specifically be an HBase cluster, where HBase is a highly reliable, high-performance, column-oriented, scalable distributed database, mainly used to store unstructured and semi-structured loose data. The distributed database cluster mentioned in the present application may also be other distributed database clusters than HBase, which are suitable for the distributed database cluster access method.

In order to solve the problem that the existing access manner cannot simultaneously meet the security, timeliness and effectiveness of cross-cluster access of a user to a distributed database cluster, the application provides an embodiment of a distributed database cluster access method implemented by applying an intermediate service layer, wherein the intermediate service layer may be a server or a server cluster, in a specific example, the intermediate service layer may be composed of a plurality of application processes, referring to fig. 1, and the distributed database cluster access method specifically includes the following contents:

step 100: and acquiring a distributed database cluster access request sent by the target user passing the identity authentication, wherein the distributed database cluster access request comprises cluster identifiers of a plurality of target distributed database clusters.

In one or more embodiments of the present application, the user may specifically refer to an application server corresponding to a tenant of the distributed database cluster. The target user refers to a user of the distributed database cluster access request currently received and processed by the intermediate service layer.

Step 200: if the target distributed database clusters are judged to be cross-user clusters which are authorized to be accessed by the target users, locally searching and obtaining connection threads corresponding to the target distributed database clusters according to cluster identifiers of the target distributed database clusters, wherein the connection threads are pre-created based on configuration files corresponding to the distributed database clusters and used for kerberos security authentication, the configuration files are pre-obtained based on a combined authentication file stored locally, and the combined authentication file is used for storing a one-to-one correspondence between the cluster identifiers of the distributed database clusters and authentication configuration information.

In step 200, cross-user cluster means that the owner of the current cluster is not the target user who issued the distributed database cluster access request, i.e., the target user does not have the right to write to the cluster. If the target distributed database cluster is a cross-user cluster which is authorized to be accessed by the target user, the target user has the right to access the cluster although the target user does not have the right to write operation to the cluster.

The authentication configuration information may be an authentication address, so that the intermediate service layer obtains the configuration file of the corresponding distributed database cluster from the authentication address in advance.

In addition, the data stored locally in step 200 refers to data stored locally in the intermediate service layer or data that can be acquired locally by the intermediate service layer, and is specifically set according to the actual application situation. However, in a more preferred manner, in the distributed database cluster access method provided in the embodiment of the present application, before the target user accesses the distributed database clusters, in order to quickly access each cluster, HBase long connections to multiple clusters are maintained locally in advance in the intermediate service layer, but if krb files of multiple clusters are loaded in sequence, the configuration files are overlaid with each other, and after authentication of the latter cluster is successful, connection of the former cluster is failed. Therefore, through research, in order to access a plurality of clusters simultaneously, so that configuration files of each cluster do not cover each other, in the intermediate service layer provided in the application, authentication files related to kerberos such as krb5.Conf and the like need to be combined first. And extracting different authentication configurations (such as kdc addresses) among the clusters, combining the configurations of the clusters into a file through a certain organization rule, and loading the file into a memory variable, so that the authentication addresses of the clusters can be obtained simultaneously in one process. Meanwhile, since the authentication ticket TGT has an authentication validity period, before the TGT fails, the ticket needs to be refreshed.

In one or more embodiments of the present application, the kerberos security authentication file may specifically refer to a krb file in the kerberos security authentication. In the Hadoop ecosystem, in order to prevent the data of the cluster from being tampered by malicious users, authorities provide a kerberos authentication mechanism to ensure the security of the cluster. The kerberos is an identity authentication protocol based on symmetric key technology, which is mainly used for identity authentication of computer networks, and is characterized in that a user can access a plurality of services (HDFS, HBase, etc.) by means of a bill (TGT) obtained by authentication by only inputting identity authentication information once. The protocol is quite secure due to the shared secret key established between each Client and Service.

The Kerberos authentication-related profile contains: a user ticket file, a krb file, etc., wherein krb file stores necessary information for this communication area, such as the location of a KDC (key distribution center), etc.

Step 300: and accessing the target distributed database clusters by the target users based on the corresponding connection threads of the target distributed database clusters.

In step 300, each upper layer tenant application establishes a unified access connection to the HBase through the intermediate service layer of the present invention, and the intermediate service layer can route to each cluster through different cluster connections by receiving user requests for accessing different clusters, thereby obtaining access data.

As can be seen from the above description, the distributed database cluster access method provided by the embodiment of the present application can effectively improve the security and the effectiveness of cross-cluster access of a user to a distributed database cluster, and can realize access to multiple clusters at the same time, ensure the effectiveness of cross-cluster access of a user to multiple distributed database clusters, effectively improve the reliability of cross-cluster access of a user to a distributed database cluster, effectively reduce the maintenance cost of cross-cluster access, and further improve the cluster access efficiency by storing a configuration file locally, thereby being capable of safely, quickly and effectively solving the problem that cross-cluster data cannot be shared, improving the reliability and the intelligentization degree of the operation of the distributed database clusters in enterprises, and effectively improving the user experience of the cluster access user.

In order to store configuration files that are not mutually overlapped locally in the intermediate service layer in advance to ensure the availability of cross-cluster access to multiple distributed database clusters by a user, in one embodiment of the distributed database cluster access method provided in the present application, referring to fig. 2, the following are specifically included before step 100 of the distributed database cluster access method:

Step 011: and obtaining the kerberos security authentication files corresponding to each distributed database cluster respectively.

Step 012: and respectively extracting authentication configuration information corresponding to each distributed database cluster from each kerberos security authentication file.

Step 013: and storing the one-to-one correspondence between each authentication configuration information and the cluster identifier of each distributed database cluster into a combined authentication file.

Step 014: and respectively acquiring configuration files corresponding to the distributed database clusters based on the authentication configuration information in the combined authentication file.

Step 015: and establishing long connection with each distributed database cluster according to the configuration files corresponding to each distributed database cluster, wherein each long connection corresponds to one connection thread.

Step 016: and updating the authentication ticket TGT corresponding to each distributed database cluster at regular time and used for performing kerberos security authentication.

It will be appreciated that the authentication ticket TGTTGT is used to prove to the KDC service on the domain controller that the user has been authenticated by the other domain controllers. The TGT is encrypted by the KRBTGT cipher hash and can be decrypted by any KDC service in the domain.

As can be seen from the above description, the distributed database cluster access method provided by the embodiment of the present application can further achieve access to multiple clusters at the same time, ensure validity of cross-cluster access of a user to multiple distributed database clusters, and effectively improve reliability of cross-cluster access of the user to the distributed database clusters.

In order to provide a specific process of user identity authentication to further improve the security of cross-cluster access of a user to a distributed database cluster, in one embodiment of the distributed database cluster access method provided in the present application, referring to fig. 3, before step 100 of the distributed database cluster access method, the method further specifically includes the following contents:

step 021: and generating a symmetric key by applying a symmetric encryption algorithm, and sending the symmetric key to a preset configuration center.

Step 022: and generating key objects of the asymmetric private key and the public key corresponding to each user by applying an asymmetric encryption algorithm, and encrypting each key object by applying the symmetric encryption algorithm.

Step 023: storing each encrypted key object into a relational database, so that after receiving a registration request sent by each user, the configuration center obtains the encrypted key object corresponding to the target user from the relational database, and sends a public key plaintext, an encrypted symmetric key and an encrypted private key object corresponding to each key object to each corresponding user, so that each user decrypts the encrypted symmetric key based on the obtained public key plaintext, decrypts the private key object based on the corresponding obtained symmetric key, and correspondingly obtains a private key for identity authentication.

It is to be understood that the specific example of the relational database may be MySQL database, which is used to store configuration information such as authorization information and cluster information. The configuration center is used for managing key objects of all tenants on the intermediate service layer.

As can be seen from the above description, the distributed database cluster access method provided by the embodiment of the present application can further improve the security and effectiveness of cross-cluster access of the distributed database clusters by the user, and the symmetric key is one-time-pad, and the private key is protected by using the dynamic key, so that the operation and maintenance personnel cannot obtain the private key of the tenant.

In order to provide a specific process of access authorization query, so as to further improve the security of cross-cluster access of a user to a distributed database cluster, in one embodiment of the distributed database cluster access method provided in the present application, the distributed database cluster access request further includes a user identifier of the target user, see fig. 4, and after step 100 and before step 200 of the distributed database cluster access method further specifically includes the following contents:

step 031: and searching authorized user identifiers corresponding to the cluster identifiers of the target distributed database clusters from an authority control table arranged in a relational database.

Step 032: judging whether the authorized user identifications corresponding to the cluster identifications of the target distributed database clusters all contain the user identifications of the target users, if so, judging that the target distributed database clusters are cross-user clusters which are authorized to be accessed by the target users.

From the above description, the distributed database cluster access method provided by the embodiment of the application can effectively improve the reliability and efficiency of inquiring whether the current user has authority to access the cluster, and can further improve the security and effectiveness of cross-cluster access of the user to the distributed database cluster.

In order to solve the problem that the existing access manner cannot simultaneously satisfy the security, timeliness and effectiveness of cross-cluster access of a user to a distributed database cluster, the application provides an embodiment of an intermediate service layer for implementing all or part of contents in the distributed database cluster access method, where the intermediate service layer may be a server or a server cluster, and in a specific example, the intermediate service layer may be composed of a plurality of application processes, see fig. 5, and specifically includes:

The request receiving module 10 is configured to obtain a distributed database cluster access request sent by a target user that has passed identity authentication, where the distributed database cluster access request includes cluster identifiers of multiple target distributed database clusters.

The thread invoking module 20 is configured to, if it is determined that each target distributed database cluster is a cross-user cluster that is authorized for the target user to access, locally find, according to a cluster identifier of each target distributed database cluster, a connection thread corresponding to each target distributed database cluster, where each connection thread is pre-created based on a configuration file for kerberos security authentication corresponding to each distributed database cluster, and each configuration file is pre-obtained based on a merged authentication file stored locally, where the merged authentication file is used to store a one-to-one correspondence between a cluster identifier of each distributed database cluster and authentication configuration information.

And the cluster access module 30 is configured to implement access of the target user to each target distributed database cluster based on the corresponding connection thread of each target distributed database cluster.

As can be seen from the above description, the intermediate service layer provided by the embodiment of the present application can effectively improve the security and effectiveness of cross-cluster access of a user to a distributed database cluster, and can realize access to multiple clusters at the same time, ensure the validity of cross-cluster access of a user to multiple distributed database clusters, effectively improve the reliability of cross-cluster access of a user to a distributed database cluster, and effectively reduce the maintenance cost of cross-cluster access, and further improve the cluster access efficiency by storing a configuration file locally, thereby being capable of safely, quickly and effectively solving the problem that cross-cluster data cannot be shared, improving the reliability and intelligentization degree of the operation of the distributed database cluster in an enterprise, and effectively improving the user experience of the cluster access user.

In order to store configuration files that are not mutually covered locally in the intermediate service layer in advance to ensure the availability of cross-cluster access of the user to the plurality of distributed database clusters, in one embodiment of the intermediate service layer provided in the present application, referring to fig. 6, the intermediate service layer further specifically includes the following contents:

A thread creation module 01, the thread creation module 01 being configured to perform the following:

As can be seen from the above description, the intermediate service layer provided in the embodiments of the present application can further realize access to multiple clusters at the same time, ensure the validity of cross-cluster access of a user to multiple distributed database clusters, and effectively improve the reliability of cross-cluster access of the user to the distributed database clusters.

In order to provide a specific process of user identity authentication, so as to further improve the security of cross-cluster access of a user to a distributed database cluster, in one embodiment of an intermediate service layer provided in the present application, referring to fig. 7, the intermediate service layer further specifically includes the following contents:

an identity authentication module 02, the identity authentication module 02 is configured to perform the following:

As can be seen from the above description, the intermediate service layer provided in the embodiments of the present application can further improve the security and effectiveness of cross-cluster access of a user to a distributed database cluster, and the symmetric key is one-time-pad, and the dynamic key is used to protect the private key, so that an operation and maintenance person cannot obtain the private key of the tenant.

In order to provide a specific process of access authorization query, so as to further improve the security of cross-cluster access of users to a distributed database cluster, in one embodiment of the intermediate service layer provided in the present application, the distributed database cluster access request further includes a user identifier of the target user, see fig. 8, and the intermediate service layer further specifically includes the following contents:

a rights inquiry module 03, the rights inquiry module 03 is configured to perform the following:

From the above description, the intermediate service layer provided in the embodiment of the present application can effectively improve reliability and efficiency of querying whether the current user has permission to access the cluster, and can further improve security and effectiveness of cross-cluster access of the user to the distributed database cluster.

In order to further explain the scheme, the application further provides a specific application example for realizing the distributed database cluster access method by applying the distributed database cluster access system, the application example relates to the Internet big data technology, a system for realizing HBase cross-cluster authorized access and a method for realizing HBase cross-cluster authorized access are provided, the problem of cluster security authentication is solved by introducing a kerberos security authentication mechanism, meanwhile, a middle service layer is packaged, the problem of authentication failure of accessing a plurality of HBase clusters by a single process is overcome in the middle service layer, a multi-tenant authentication and authentication mechanism is provided, and connection for accessing the plurality of clusters is maintained, so that the system and the method for realizing HBase cross-cluster authorized access are provided, and the problem that cross-cluster data cannot be shared is safely and at low cost.

In order to achieve the purpose of preventing the data of the cluster from being tampered by malicious users, kerberos is used on the HBase cluster for security authentication. However, because Kerberos authentication addresses of different HBase clusters are different, krb files are loaded as one memory variable when connected to the HBase clusters, so each Client process can only access one HBase cluster using the original krb files.

In the intermediate service layer, in order to quickly access each cluster, the intermediate service layer maintains HBase long connections to a plurality of clusters in advance, but if krb files of a plurality of clusters are loaded in sequence, configuration files are mutually covered, and after the authentication of the latter cluster is successful, the connection of the former cluster is invalid. Therefore, through research, in order to access a plurality of clusters simultaneously, so that configuration files of each cluster do not cover each other, in the intermediate service layer provided in the application, authentication files related to kerberos such as krb5.Conf and the like need to be combined first. And extracting different authentication configurations (such as kdc addresses) among the clusters, combining the configurations of the clusters into a file through a certain organization rule, and loading the file into a memory variable, so that the authentication addresses of the clusters can be obtained simultaneously in one process. Meanwhile, since the authentication ticket TGT has an authentication validity period, before the TGT fails, the ticket needs to be refreshed.

The upper layer tenant application establishes unified access connection to the HBase through the intermediate service layer of the application instance, and the intermediate service layer can route to each cluster through different cluster connections by receiving user requests for accessing different clusters to obtain access data. Considering that the operation flow involves data transmission between different tenants and the intermediate service layer, the intermediate service layer provides authentication and authorization mechanisms. In order to reduce the key maintenance cost of each tenant application operation and maintenance personnel, the authentication key of the application instance is generated in a mode that a configuration center issues the key uniformly and a dynamic key protects a private key.

Referring to fig. 9, the distributed database cluster access system specifically includes: the system comprises an initialization module 1, an intermediate service layer tenant authentication module 2, an authorization execution module 3, a cluster connection creation and refreshing module 4 and a data access module 5.

The initialization module 1 is responsible for assigning tenants to each application, creating an authorization control table in a relational database, and the like. The intermediate service layer tenant authentication module 2 is responsible for verifying the authentication of each tenant on the intermediate service layer and the key distribution. The authorization execution module 3 is responsible for executing the authorization operations of the data tables between different applications. The cluster connection creation and refreshing module 4 is responsible for ensuring that the connection is available, establishing the connection with each cluster through the authentication of the kerberos and executing the refreshing operation before the ticket expires. The data access module 5 is responsible for providing data access capabilities, e.g. enabling tenant B to access data of tenant a across clusters using its own applications.

(II) referring to FIG. 10, the initialization module 1 performs the following initialization procedure:

step 201: allocating tenants for each application: the initialization module 1 in the intermediate service layer allocates tenants for each application. Assuming that the tenant corresponding to the upstream application is tenant a, and the tenant corresponding to the downstream application is tenant B.

Step 202: newly-built permission control table: a new permission control table is built in the relational database for recording the permission information between applications, for example: table name hbase_acl_info.

The fields contain: application name app_name, authorized table name table_name, authorized application authorized_app_name, and authorized operation operations, etc.

Step 203: preparing cluster configuration and kerberos security authentication related configuration files: assuming that data of an application of the tenant a is stored on the HBase cluster A1, data of an application of the tenant B is stored on the HBase cluster B1, relevant configuration information, such as HBase-site.xml, of the HBase cluster A1 and the HBase cluster B1 needs to be prepared in advance, and configuration files (krb 5. Conf) and the like, which are respectively related to security authentication, of the cluster user userA and the cluster user userB are prepared.

Step 204: and starting a main service process deployed on each server, establishing connection, and caching an authorization table in a memory. The authorization table information in the relational database is cached in the memory, so that the access efficiency is improved.

(III) each tenant accesses the HBase cluster through the connection provided by the intermediate service layer, in order to prevent the data of each tenant from being modified by other tenants, the intermediate service layer provides an authentication and authentication mechanism, and referring to FIG. 11, the step flow of the intermediate service layer tenant authentication module 2 for each tenant private key propagation is as follows:

step 301: generating a symmetric algorithm key: after the main service process is started, the intermediate service layer generates a symmetric algorithm key through a symmetric encryption algorithm (such as triple des and the like), stores the key in a file in an object serialization mode, and then propagates the key file to a configuration center. After each restart of the main service process, the key is regenerated to achieve the one-time pad effect.

Step 302: generating an asymmetric algorithm private key and a public key object: the intermediate service layer generates an asymmetric algorithm private key and a public key object for each tenant by using an asymmetric encryption algorithm, encrypts by using the symmetric algorithm key and stores the encrypted object in a relational database, and each tenant performs identity authentication by using the asymmetric key.

Step 303: initiate user request and pull key object: the configuration center sends a request, pulls key objects of all tenants from the relational database, decrypts the key objects by using a symmetric algorithm and stores the key objects.

Step 304: initiating a registration request: the application server of the tenant initiates a registration request to the configuration center, and the configuration management center automatically issues a new key according to registration information, and the configuration center is used for storing a key file, so that the advantages are that: all configurations are registered in a configuration center, and once missing applications report errors, the missing applications can be found in time.

Step 305: distributing a key: the configuration center distributes the public key plaintext of each tenant, the symmetric key encrypted by the public key through the symmetric algorithm, and each private key object encrypted by the symmetric key to all servers of each tenant.

Step 306: identity authentication is realized: after each tenant application server receives the data, the public key is used by itself to decrypt and obtain the symmetric key, and then the symmetric key is used to decrypt and obtain the private key, so that the identity authentication is realized to access the intermediate service layer.

The symmetric algorithm has the advantages that: the encryption speed is high, and the method is suitable for communication data transmission, and has the defects that the content of the secret key used by both encryption parties is consistent, and one party can be cracked after being leaked. Asymmetric key advantage: the private key is encrypted and decrypted, the public key can be disclosed, the public key and the private key are different in content, and the private key can not be restored only by revealing the public key. The private key is used for identity authentication, which has the following disadvantages: the encryption speed is low, and the method is suitable for encrypting a small amount of information. According to the scheme, the characteristics of the big data cluster are combined, the symmetric key is encrypted at one time, and the private key is protected by using the dynamic key, so that operation and maintenance personnel cannot acquire the private key of the tenant.

The authorization executing module executes the following authorization flow:

tenant A authorizes the table of tenant A to access to tenant B through the grantSelect authorization interface by providing the following authorization information, and the service process of the intermediate service layer writes the authorization information into the authority control table hbase_acl_info shown in Table 1.

TABLE 1

Data writing-in party	Data reading device	Table name	Access mode
				Tenant A	Tenant B	A_TABLE_1	R (read-only)
Tenant A	Tenant B	A_TABLE_2	R (read-only)

(fifth), referring to fig. 12, after the initialization module 1 and the authorization execution module 3 execute, the specific process of the distributed database cluster access method implemented by the application example of the present application is as follows:

step 401: request: the application server of the upper layer sends a request to the application process in the intermediate service layer.

Step 402: access grant table: and the application process acquires the authorization information of the MySQL service from the authority control table of the relational database according to the requested application name and table name, and judges whether the access authority exists according to a certain rule.

Step 403: refreshing: if the right to access the HBase exists, the safe connection with the HBase cluster A1 and the HBase cluster B1 is established through the cluster connection creation and refreshing module 4, and the connected bill is refreshed at fixed time.

Step 404: results: the designated connection is acquired by the data access module 5 and data access is provided.

(six) referring to fig. 13, after the application server is started, the authority access rule is executed in step 402 as follows:

step 501: the upper layer application sends a request with application name a' and the TABLE name to be accessed a_table_1.

Step 502: whether the application a' is the owner wner of the a_table_1 TABLE: and the main service process on each server receives the user request, and the main process newly adds a checkReadAccess () method according to the application name and the table name of the request to judge the authority. For example, the application name received in the request is a ', the TABLE name to be accessed is a_table_1, it is determined whether the application a' is the owner of the requested TABLE a_table_1, and if so, step 503 is performed; if not, step 504 is performed.

Step 503: returning the connection thread of the corresponding cluster of the application A', and responding to the request: the application is the owner of the table, and the main thread should return to the connected thread of the corresponding cluster of the application, and can respond to the request and return the result to the application.

Step 504: access rights control TABLE, whether application a' has access rights to the a_table_1 TABLE: if the application is not the owner of the table owner, the access control table hbase_acl_info determines if the application has access to the table. If the authorization record exists in the authority control table in the table, executing step 505; otherwise, step 506 is performed.

Step 505: returning the connection thread of the corresponding cluster of the application to which the A_TABLE_1 TABLE belongs, and responding to the request: the application is not the owner of the TABLE, but is authorized to have access to the TABLE, and the main thread should return to the connection thread of the corresponding cluster of the application to which the a_table_1 belongs, and can respond to the request and return the result to the application.

Step 506: throwing an anomaly of "403-rights deficient": application a' is not the owner of the table and is unauthorized, does not have permission to access the table, and throws the exception of "403-not-sufficient".

(seventh) referring to fig. 14, the flow of the cluster connection creation and refresh module 4 creation and maintenance thread is as follows, in which long connections accessing different clusters are maintained in the main service process, each connection being created and maintained by a separate thread.

Step 601: krb5 etc. configuration files of each cluster are combined: the krb5.Conf files of the respective clusters already prepared in the initialization module 1 are combined so that multiple HBase clusters can be accessed within a single process.

Step 602: sequentially loading HBase configuration files of each cluster, acquiring access ticket TGT of the cluster, and creating long connection with the cluster: and sequentially loading HBase configuration files of the clusters, acquiring access ticket TGT of each cluster, and creating long connection with the cluster. For example, loading a configuration file of the HBase cluster A1, acquiring bills from the user A to the HBase cluster A1, and creating long connection to the HBase cluster A1; loading a configuration file of the HBase cluster B1, acquiring a bill from the user B to the HBase cluster B1, creating a long connection to the HBase cluster B1, and the like.

Step 603: and refreshing the bill before the bill expires: several hours before the ticket of each cluster expires, a refresh ticket operation is performed.

As can be seen from the above description, the distributed database cluster access system and method provided in the embodiments of the present application solve the problem of cluster security authentication through a kerberos security authentication mechanism, meanwhile, encapsulate a layer of intermediate service layer, and overcome the problem of authentication failure of accessing multiple HBase clusters by a single process in the intermediate service layer, provide a multi-tenant authentication and authentication mechanism, maintain the connection accessing multiple clusters, so that users of different clusters can access tables and data in other clusters HBase across clusters, and solve the problem that cross-cluster data cannot be shared by a safe, general and low-cost method.

In order to solve the problem that the existing distributed database cluster access mode cannot simultaneously meet the safety, timeliness and effectiveness of cross-cluster access of a user to a distributed database cluster, the application provides an embodiment of an electronic device for realizing all or part of contents in the distributed database cluster access method, wherein the electronic device specifically comprises the following contents:

Fig. 15 is a schematic block diagram of a system configuration of the electronic device 9600 of the embodiment of the present application. As shown in fig. 15, the electronic device 9600 may include a central processor 9100 and a memory 9140; the memory 9140 is coupled to the central processor 9100. Notably, this fig. 15 is exemplary; other types of structures may also be used in addition to or in place of the structures to implement telecommunications functions or other functions.

In one embodiment, the distributed database cluster access functionality may be integrated into the central processor. Wherein the central processor may be configured to control:

As can be seen from the above description, the electronic device provided by the embodiment of the present application can effectively improve the security and effectiveness of cross-cluster access of a user to a distributed database cluster, and can realize access to multiple clusters at the same time, ensure the effectiveness of cross-cluster access of a user to multiple distributed database clusters, effectively improve the reliability of cross-cluster access of a user to a distributed database cluster, and effectively reduce the maintenance cost of cross-cluster access, and further improve the cluster access efficiency by storing a configuration file locally, thereby being capable of safely, quickly and effectively solving the problem that cross-cluster data cannot be shared, improving the reliability and intelligentization degree of the operation of the distributed database cluster in an enterprise, and effectively improving the user experience of the cluster access user.

In another embodiment, the intermediate service layer may be configured separately from the central processor 9100, for example, the intermediate service layer may be configured as a chip connected to the central processor 9100, and the distributed database cluster access function is implemented under the control of the central processor.

As shown in fig. 15, the electronic device 9600 may further include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It is noted that the electronic device 9600 need not include all of the components shown in fig. 15; in addition, the electronic device 9600 may further include components not shown in fig. 15, and reference may be made to the related art.

As shown in fig. 15, the central processor 9100, sometimes referred to as a controller or operational control, may include a microprocessor or other processor device and/or logic device, which central processor 9100 receives inputs and controls the operation of the various components of the electronic device 9600.

The memory 9140 may be, for example, one or more of a buffer, a flash memory, a hard drive, a removable media, a volatile memory, a non-volatile memory, or other suitable device. The information about failure may be stored, and a program for executing the information may be stored. And the central processor 9100 can execute the program stored in the memory 9140 to realize information storage or processing, and the like.

The input unit 9120 provides input to the central processor 9100. The input unit 9120 is, for example, a key or a touch input device. The power supply 9170 is used to provide power to the electronic device 9600. The display 9160 is used for displaying display objects such as images and characters. The display may be, for example, but not limited to, an LCD display.

The memory 9140 may be a solid state memory such as Read Only Memory (ROM), random Access Memory (RAM), SIM card, etc. But also a memory which holds information even when powered down, can be selectively erased and provided with further data, an example of which is sometimes referred to as EPROM or the like. The memory 9140 may also be some other type of device. The memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application/function storage portion 9142, the application/function storage portion 9142 storing application programs and function programs or a flow for executing operations of the electronic device 9600 by the central processor 9100.

The memory 9140 may also include a data store 9143, the data store 9143 for storing data, such as contacts, digital data, pictures, sounds, and/or any other data used by an electronic device. The driver storage portion 9144 of the memory 9140 may include various drivers of the electronic device for communication functions and/or for performing other functions of the electronic device (e.g., messaging applications, address book applications, etc.).

The communication module 9110 is a transmitter/receiver 9110 that transmits and receives signals via an antenna 9111. A communication module (transmitter/receiver) 9110 is coupled to the central processor 9100 to provide input signals and receive output signals, as in the case of conventional mobile communication terminals.

Based on different communication technologies, a plurality of communication modules 9110, such as a cellular network module, a bluetooth module, and/or a wireless local area network module, etc., may be provided in the same electronic device. The communication module (transmitter/receiver) 9110 is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130 to provide audio output via the speaker 9131 and to receive audio input from the microphone 9132 to implement usual telecommunications functions. The audio processor 9130 can include any suitable buffers, decoders, amplifiers and so forth. In addition, the audio processor 9130 is also coupled to the central processor 9100 so that sound can be recorded locally through the microphone 9132 and sound stored locally can be played through the speaker 9131.

The embodiments of the present application further provide a computer readable storage medium capable of implementing all the steps in the distributed database cluster access method in the above embodiments, where the computer readable storage medium stores a computer program, and when the computer program is executed by a processor, the computer program implements all the steps in the distributed database cluster access method in which the execution subject in the above embodiments is a server or a client, for example, the processor implements the following steps when executing the computer program:

As can be seen from the above description, the computer readable storage medium provided by the embodiments of the present application can effectively improve the security and effectiveness of cross-cluster access of a user to a distributed database cluster, and can realize access to multiple clusters at the same time, ensure the effectiveness of cross-cluster access of a user to multiple distributed database clusters, effectively improve the reliability of cross-cluster access of a user to a distributed database cluster, and effectively reduce the maintenance cost of cross-cluster access, and further improve the cluster access efficiency by storing a configuration file locally, thereby safely, quickly and effectively solving the problem that cross-cluster data cannot be shared, improving the reliability and intelligentization degree of the operation of the distributed database cluster in an enterprise, and effectively improving the user experience of a cluster access user.

It will be apparent to those skilled in the art that embodiments of the present invention may be provided as a method, apparatus, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (devices), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

The principles and embodiments of the present invention have been described in detail with reference to specific examples, which are provided to facilitate understanding of the method and core ideas of the present invention; meanwhile, as those skilled in the art will have variations in the specific embodiments and application scope in accordance with the ideas of the present invention, the present description should not be construed as limiting the present invention in view of the above.

Claims

1. A distributed database cluster access method, comprising:

acquiring authentication configuration information corresponding to configuration files for performing kerberos security authentication, wherein the configuration files correspond to each distributed database cluster respectively; storing the one-to-one correspondence between each authentication configuration information and cluster identifiers of each distributed database cluster into a combined authentication file; acquiring configuration files corresponding to the distributed database clusters respectively based on the authentication configuration information in the combined authentication file; storing the one-to-one correspondence between the configuration files and cluster identifications of the distributed database clusters to a local place; establishing long connection with each distributed database cluster according to configuration files corresponding to each distributed database cluster, wherein each long connection corresponds to one connection thread; and updating the authentication ticket TGT which is used for performing kerberos security authentication and corresponds to each distributed database cluster at regular time;

Acquiring a distributed database cluster access request sent by a target user passing identity authentication, wherein the distributed database cluster access request comprises cluster identifiers of a plurality of target distributed database clusters, the user refers to an application server corresponding to a tenant of the distributed database cluster, and the target user refers to a user of the distributed database cluster access request currently received and processed by an intermediate service layer;

if the target distributed database clusters are judged to be cross-user clusters which are authorized to be accessed by the target users, locally searching and obtaining connection threads corresponding to the target distributed database clusters according to cluster identifiers of the target distributed database clusters, wherein the connection threads are pre-created based on configuration files corresponding to the distributed database clusters and used for kerberos security authentication, the configuration files are pre-acquired based on a locally-stored combined authentication file, the combined authentication file is used for storing a one-to-one correspondence between the cluster identifiers of the distributed database clusters and authentication configuration information, and the authentication configuration information is an authentication address, so that an intermediate service layer obtains the configuration files of the corresponding distributed database clusters from the authentication address in advance;

2. The distributed database cluster access method according to claim 1, further comprising, before the obtaining the distributed database cluster access request sent by the authenticated target user:

3. The distributed database cluster access method according to claim 1, wherein the distributed database cluster access request further includes a user identifier of the target user;

4. An intermediate service layer, comprising:

a thread creation module for performing the following:

and updating the authentication ticket TGT which is used for performing kerberos security authentication and corresponds to each distributed database cluster at regular time;

the system comprises a request receiving module, a service layer and a service layer, wherein the request receiving module is used for obtaining a distributed database cluster access request sent by a target user passing identity authentication, the distributed database cluster access request comprises cluster identifiers of a plurality of target distributed database clusters, the user refers to an application server corresponding to a tenant of the distributed database cluster, and the target user refers to a user of the distributed database cluster access request currently received and processed by the intermediate service layer;

The thread invoking module is used for locally searching and obtaining connection threads corresponding to the target distributed database clusters according to the cluster identification of the target distributed database clusters if the target distributed database clusters are judged to be cross-user clusters which are authorized to be accessed by the target users, wherein the connection threads are pre-created based on the configuration files corresponding to the distributed database clusters and used for kerberos security authentication, the configuration files are pre-obtained based on a combined authentication file stored locally, the combined authentication file is used for storing a one-to-one correspondence between the cluster identification of the distributed database clusters and authentication configuration information, and the authentication configuration information is an authentication address, so that an intermediate service layer obtains the configuration files of the corresponding distributed database clusters from the authentication address in advance;

5. The intermediary service layer of claim 4, further comprising: the identity authentication module is used for executing the following contents:

6. The intermediary service layer according to claim 4, wherein the distributed database cluster access request further comprises a user identifier of the target user;

7. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the distributed database cluster access method of any of claims 1 to 3 when the program is executed by the processor.

8. A computer readable storage medium having stored thereon a computer program, which when executed by a processor implements the distributed database cluster access method of any of claims 1 to 3.