CN111737741A

CN111737741A - Distributed database cluster access method and intermediate service layer

Info

Publication number: CN111737741A
Application number: CN202010564789.9A
Authority: CN
Inventors: 刘雪晶; 林立成; 翁晓俊; 王之乐
Original assignee: Industrial and Commercial Bank of China Ltd ICBC
Current assignee: Industrial and Commercial Bank of China Ltd ICBC
Priority date: 2020-06-19
Filing date: 2020-06-19
Publication date: 2020-10-02
Anticipated expiration: 2040-06-19
Also published as: CN111737741B

Abstract

The embodiment of the application provides a distributed database cluster access method and an intermediate service layer, wherein the method comprises the following steps: the method comprises the steps that connection threads corresponding to target distributed database clusters are obtained through local searching according to cluster identifications of the target distributed database clusters, each connection thread is created based on each configuration file for kerberos safety authentication, each configuration file is obtained based on a local combined authentication file, and the combined authentication files are used for storing one-to-one correspondence between the cluster identifications and authentication configuration information; and realizing the access of the target user to each target distributed database cluster based on the connection thread. According to the method and the device, the user can access the distributed database clusters simultaneously, so that the safety and the timeliness of cross-cluster access of the user to the distributed database clusters are effectively improved, and the effectiveness and the reliability of cross-cluster access of the user to the distributed database clusters are improved simultaneously.

Description

Distributed database cluster access method and intermediate service layer

Technical Field

The application relates to the technical field of data processing, in particular to a distributed database cluster access method and an intermediate service layer.

Background

With the rapid development of information technology, the structured and semi-structured data including characters, images, videos and the like realize exponential growth. The traditional database is difficult to store and analyze the content of the data, so the industry generally uses big data technology such as Hadoop ecosystem to process business. The distributed database such as HBase and the like is used as an important component of an ecosphere such as Hadoop and the like, has the main characteristics of supporting real-time storage and query of mass data, and has the characteristics of high reliability, high performance, orientation, scalability and the like, so that the distributed database is widely used in industries such as electronic commerce, Internet of things and the like.

As more and more enterprises or users use distributed databases, there is a significant problem within each enterprise: each sub-department or application has its own application server, and the application servers are relatively independent, and in a cross-cluster scene, different cluster user data cannot be shared, and the defect that data cannot be shared exists among independent systems.

At present, two common solutions exist, one is to use a non-secure authentication mode in a distributed database cluster of each service system, and a cluster user interacts with the cluster without verification, so that only configuration files of different clusters are sequentially loaded, and different connection examples are created, and data of different clusters can be accessed. However, this scheme has a security problem, and there is a risk that data is tampered by a malicious user. Another approach may take the form of data-handling, i.e., cross-cluster data copying. However, as the amount of traffic data increases further, more and more data are copied, which significantly increases the operation cost and causes data redundancy; meanwhile, when the clusters cannot communicate, the problem of cluster mutual trust also needs to be considered, so that cross-cluster data copying has certain limitation, and the difficulty and timeliness of data sharing are greatly increased. The other method is to perform security verification on cluster users, but in the authentication process, authentication files of multiple clusters need to be loaded in sequence, which may cause the clusters to overlap configuration files, that is, after the authentication of a subsequent cluster succeeds, the connection of the previous cluster fails, so that the requirement that the cluster users access multiple clusters simultaneously cannot be met simultaneously, and the effectiveness of the distributed database cluster is reduced. That is, in any of the above-described access methods of the prior art, the security, timeliness, and effectiveness of the cross-cluster access of the user to the distributed database cluster cannot be simultaneously satisfied.

Disclosure of Invention

Aiming at the problems in the prior art, the application provides a distributed database cluster access method and an intermediate service layer, which can enable a user to simultaneously access a plurality of distributed database clusters, so that the safety and the effectiveness of cross-cluster access of the user to the distributed database clusters can be effectively improved, and the effectiveness and the reliability of cross-cluster access of the user to the distributed database clusters can be improved.

In order to solve the technical problem, the application provides the following technical scheme:

in a first aspect, the present application provides a distributed database cluster access method, including:

acquiring a distributed database cluster access request sent by a target user who passes identity authentication, wherein the distributed database cluster access request comprises cluster identifications of a plurality of target distributed database clusters;

if the target distributed database clusters are judged to be cross-user clusters authorized to be accessed by the target user, local searching is performed according to cluster identifiers of the target distributed database clusters to obtain connecting threads corresponding to the target distributed database clusters, wherein the connecting threads are pre-created based on configuration files for kerberos security authentication corresponding to the distributed database clusters, the configuration files are pre-acquired based on a combined authentication file stored locally, and the combined authentication file is used for storing a one-to-one correspondence relationship between the cluster identifiers and authentication configuration information of the distributed database clusters;

and realizing the access of the target user to each target distributed database cluster based on the connection thread corresponding to each target distributed database cluster.

Further, before the obtaining the distributed database cluster access request sent by the target user who has passed the identity authentication, the method further includes:

acquiring authentication configuration information corresponding to configuration files for carrying out kerberos security authentication, which correspond to all distributed database clusters respectively;

storing the one-to-one correspondence between each authentication configuration information and the cluster identification of each distributed database cluster into a combined authentication file;

respectively acquiring configuration files corresponding to the distributed database clusters based on the authentication configuration information in the combined authentication file;

storing the one-to-one correspondence between each configuration file and the cluster identifier of each distributed database cluster to the local;

respectively establishing long connections with the distributed database clusters according to configuration files corresponding to the distributed database clusters, wherein each long connection corresponds to a connection thread;

and updating the authentication ticket TGT which is used for carrying out the kerberos security authentication and corresponds to each distributed database cluster at regular time.

Further, before the obtaining of the distributed database cluster access request sent by the target user who has passed the identity authentication, the method further includes:

generating a symmetric key by applying a symmetric encryption algorithm, and sending the symmetric key to a preset configuration center;

generating key objects of asymmetric private keys and public keys corresponding to the users by applying an asymmetric encryption algorithm, and encrypting the key objects by applying the symmetric encryption algorithm;

storing each encrypted key object into a relational database, so that the configuration center obtains the encrypted key object corresponding to the target user from the relational database after receiving a registration request sent by each user, and sends a public key plaintext, an encrypted symmetric key and an encrypted private key object which respectively correspond to each key object to each corresponding user, so that each user respectively decrypts the encrypted symmetric key based on the obtained public key plaintext and decrypts the private key object based on the obtained symmetric key, and correspondingly obtains a private key for identity authentication.

Further, the access request of the distributed database cluster also comprises a user identifier of the target user;

correspondingly, before the local search is performed according to the cluster identifier of each target distributed database cluster to obtain the configuration file corresponding to each target distributed database cluster, the method further includes:

searching authorized user identifications corresponding to the cluster identifications of the target distributed database clusters from an authority control table arranged in a relational database;

and judging whether the authorized user identification corresponding to the cluster identification of each target distributed database cluster comprises the user identification of the target user, if so, judging that each target distributed database cluster is a cross-user cluster which is authorized to be accessed by the target user.

In a second aspect, the present application provides an intermediate service layer, comprising:

the request receiving module is used for acquiring a distributed database cluster access request sent by a target user who passes identity authentication, wherein the distributed database cluster access request comprises cluster identifications of a plurality of target distributed database clusters;

the thread calling module is used for locally searching and obtaining a connection thread corresponding to each target distributed database cluster according to a cluster identifier of each target distributed database cluster if the target distributed database cluster is judged to be a cross-user cluster authorized to be accessed by the target user, wherein each connection thread is created in advance based on a configuration file which is corresponding to each distributed database cluster and is used for kerberos safety authentication, each configuration file is acquired in advance based on a combined authentication file stored locally, and the combined authentication file is used for storing the one-to-one correspondence relationship between the cluster identifier of each distributed database cluster and authentication configuration information;

and the cluster access module is used for realizing the access of the target user to each target distributed database cluster based on the connection thread corresponding to each target distributed database cluster.

Further, still include: a thread creation module to perform the following:

Further, still include: an identity authentication module for performing the following:

correspondingly, the intermediate service layer further comprises: a permission query module for executing the following:

In a third aspect, the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor implements the distributed database cluster access method when executing the computer program.

In a fourth aspect, the present application provides a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the distributed database cluster access method described.

According to the technical scheme, the distributed database cluster access method and the intermediate service layer provided by the application comprise the following steps: acquiring a distributed database cluster access request sent by a target user who passes identity authentication, wherein the distributed database cluster access request comprises cluster identifications of a plurality of target distributed database clusters; if the target distributed database clusters are judged to be cross-user clusters authorized to be accessed by the target user, local searching is performed according to cluster identifiers of the target distributed database clusters to obtain connecting threads corresponding to the target distributed database clusters, wherein the connecting threads are pre-created based on configuration files for kerberos security authentication corresponding to the distributed database clusters, the configuration files are pre-acquired based on a combined authentication file stored locally, and the combined authentication file is used for storing a one-to-one correspondence relationship between the cluster identifiers and authentication configuration information of the distributed database clusters; the method can effectively improve the safety and the timeliness of cross-cluster access of a user to the distributed database cluster, can realize simultaneous access to a plurality of clusters, can ensure the effectiveness of cross-cluster access of the user to the plurality of distributed database clusters, can effectively improve the reliability of cross-cluster access of the user to the distributed database cluster, can effectively reduce the maintenance cost of cross-cluster access, and further improves the cluster access efficiency by storing the configuration file locally, thereby being capable of safely, cheaply, quickly and effectively solving the problem that cross-cluster data cannot be shared, improving the reliability and the intelligent degree of operation of the distributed database cluster in an enterprise, and effectively improving the user experience of cluster access users.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.

Fig. 1 is a flowchart illustrating a distributed database cluster access method in an embodiment of the present application.

Fig. 2 is a schematic flowchart illustrating a specific process from step 011 to step 016 in the distributed database cluster accessing method in the embodiment of the present application.

Fig. 3 is a schematic specific flowchart of steps 021 to 023 in the distributed database cluster access method in the embodiment of the present application.

Fig. 4 is a schematic flowchart of

steps

031 and 032 in the distributed database cluster access method in this embodiment.

Fig. 5 is a first structural diagram of an intermediate service layer in the embodiment of the present application.

Fig. 6 is a second structural diagram of an intermediate service layer in the embodiment of the present application.

Fig. 7 is a third structural diagram of an intermediate service layer in the embodiment of the present application.

Fig. 8 is a fourth structural diagram of an intermediate service layer in the embodiment of the present application.

Fig. 9 is a schematic structural diagram of a distributed database cluster access system provided in an application example of the present application.

Fig. 10 is a flowchart of initialization executed by an initialization module provided in an application example of the present application.

Fig. 11 is a flowchart of the intermediate service layer tenant authentication module provided in the application example of the present application for propagating private keys of tenants.

Fig. 12 is a schematic flowchart of a distributed database cluster access method provided in an application example of the present application.

Fig. 13 is a schematic diagram of an execution process of the right access rule provided by the application example of the present application.

Fig. 14 is a schematic flowchart of creating and maintaining a thread of the cluster connection creating and refreshing module 4 according to an application example of the present application.

Fig. 15 is a schematic structural diagram of an electronic device in an embodiment of the present application.

Detailed Description

In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

In order to solve the problem that the existing distributed database cluster access mode cannot simultaneously meet the requirements of safety, timeliness and effectiveness of cross-cluster access of a user for a distributed database cluster, the application respectively provides an embodiment of a distributed database cluster access method, an intermediate service layer for realizing the distributed database cluster access method, electronic equipment and a computer readable storage medium, and a distributed database cluster access request sent by a target user who passes identity authentication is obtained, wherein the distributed database cluster access request comprises cluster identifications of a plurality of target distributed database clusters; if the target distributed database clusters are judged to be cross-user clusters authorized to be accessed by the target user, local searching is performed according to cluster identifiers of the target distributed database clusters to obtain connecting threads corresponding to the target distributed database clusters, wherein the connecting threads are pre-created based on configuration files for kerberos security authentication corresponding to the distributed database clusters, the configuration files are pre-acquired based on a combined authentication file stored locally, and the combined authentication file is used for storing a one-to-one correspondence relationship between the cluster identifiers and authentication configuration information of the distributed database clusters; the method comprises the steps of realizing the access of target users to each target distributed database cluster based on the connection thread corresponding to each target distributed database cluster, introducing a kerberos security authentication mechanism, solving the problem of cluster security authentication, encapsulating a layer of middle service layer, overcoming the problem of authentication failure of accessing a plurality of distributed database clusters by a single process in the middle service layer, providing a multi-tenant authentication and authentication mechanism, maintaining the connection of accessing a plurality of clusters in the middle service layer, enabling users of data in different distributed database clusters to safely cross-cluster query and access tables and data in other clusters, and solving the problem that cross-cluster data cannot be mutually accessed at low cost.

Specifically, the following examples are given to illustrate the respective embodiments.

In one or more embodiments of the present application, an example of the distributed database cluster may specifically be an HBase cluster, where the HBase is a highly reliable, high-performance, column-oriented, scalable distributed database, and is mainly used to store unstructured and semi-structured loose data. The distributed database cluster mentioned in the application can also be other distributed database clusters except HBase which are applicable to the distributed database cluster access method.

In order to solve the problem that an existing access method cannot simultaneously meet the requirements of security, timeliness and effectiveness of cross-cluster access of a user to a distributed database cluster, the present application provides an embodiment of a distributed database cluster access method implemented by an application intermediary service layer, where the intermediary service layer may be a server or a server cluster, and in a specific example, the intermediary service layer may be composed of a plurality of application processes, see fig. 1, and the distributed database cluster access method specifically includes the following contents:

step 100: and acquiring a distributed database cluster access request sent by a target user passing identity authentication, wherein the distributed database cluster access request comprises cluster identifications of a plurality of target distributed database clusters.

In one or more embodiments of the present application, the user may specifically refer to an application server corresponding to a tenant of a distributed database cluster. The target user refers to a user of the distributed database cluster access request which is currently received and processed by the intermediate service layer.

Step 200: and if the target distributed database clusters are judged to be cross-user clusters authorized to be accessed by the target user, searching locally according to cluster identifiers of the target distributed database clusters to obtain connecting threads corresponding to the target distributed database clusters, wherein the connecting threads are created in advance based on configuration files for kerberos security authentication corresponding to the distributed database clusters, the configuration files are acquired in advance based on a combined authentication file stored locally, and the combined authentication file is used for storing the one-to-one correspondence relationship between the cluster identifiers and authentication configuration information of the distributed database clusters.

In step 200, the cross-user cluster means that the owner of the current cluster is not the target user who issues the access request of the distributed database cluster, that is, the target user does not have the right to perform the write operation on the cluster. And if the target distributed database cluster is a cross-user cluster which is authorized to be accessed by the target user, the target user does not have the authority of writing operation to the cluster, but has the authority of accessing the cluster.

The authentication configuration information may specifically be an authentication address, so that the intermediate service layer obtains a configuration file of the corresponding distributed database cluster from the authentication address in advance.

In addition, the data stored locally in step 200 refers to data stored locally in the intermediate service layer or data that can be acquired locally by the intermediate service layer, and is specifically set according to an actual application situation. However, in a more preferable mode, the distributed database cluster access method provided in the embodiment of the present application may be implemented, before a target user accesses a distributed database cluster, in order to quickly access each cluster, an HBase long connection to multiple clusters is locally maintained in advance in an intermediate service layer, but if krb5 files of multiple clusters are sequentially loaded, mutual coverage among configuration files may be caused, and after the authentication of the subsequent cluster is successful, the connection of the previous cluster may fail. Therefore, as a result of research, in order to simultaneously access a plurality of clusters and prevent the configuration files of the clusters from overlapping each other, in the intermediate service layer provided in the present application, it is first necessary to merge authentication files related to kerberos such as krb5. conf. The authentication configurations (such as kdc addresses) with difference among the clusters are extracted, the configurations of the clusters are combined into a file through a certain organization rule, and then the file is loaded into a memory variable, so that the authentication addresses of the clusters can be obtained simultaneously in one process. Meanwhile, the authentication ticket TGT has an authentication validity period, so that the ticket needs to be refreshed before the TGT is invalid.

In one or more embodiments of the present application, the kerberos security authentication file may specifically refer to krb5 file in kerberos security authentication. In the Hadoop ecosystem, in order to prevent data of the cluster from being tampered by malicious users, a kerberos authentication mechanism is provided by an official to guarantee the security of the cluster. The kerberos is an identity authentication protocol based on a symmetric key technology, is mainly used for identity authentication of a computer network, and is characterized in that a user can access a plurality of services (HDFS, HBase and the like) by verifying a ticket (TGT) obtained by authentication only by inputting identity authentication information once. The protocol has considerable security because a shared secret key is established between each Client and Service.

The Kerberos authentication related configuration file contains: user ticket file, krb5 file, etc., where the krb5 file would have stored therein the necessary information for this communication zone, such as the location of the KDC (key distribution center), etc.

Step 300: and realizing the access of the target user to each target distributed database cluster based on the connection thread corresponding to each target distributed database cluster.

In step 300, each upper tenant application establishes a uniform access connection to the HBase through the intermediate service layer of the present invention, and the intermediate service layer can route to each cluster through different cluster connections by receiving a user request for accessing different clusters, thereby obtaining access data.

From the above description, the distributed database cluster access method provided in the embodiments of the present application can effectively improve the security and the efficiency of the cross-cluster access of the user to the distributed database cluster, and can realize the access aiming at a plurality of clusters simultaneously, can ensure the effectiveness of cross-cluster access of a user to a plurality of distributed database clusters, and can effectively improve the reliability of the cross-cluster access of the user to the distributed database clusters, meanwhile, the maintenance cost of cross-cluster access can be effectively reduced, the cluster access efficiency is further improved by storing the configuration file locally, therefore, the problem that cross-cluster data cannot be shared can be safely, quickly and effectively solved at low cost, the reliability and the intelligent degree of operation of the distributed database cluster in an enterprise are improved, and the user experience of a cluster access user can be effectively improved.

In order to store configuration files that are not mutually covered locally in the intermediate service layer in advance, so as to ensure the effectiveness of cross-cluster access to multiple distributed database clusters by a user, in an embodiment of the distributed database cluster access method provided by the present application, referring to fig. 2, step 100 of the distributed database cluster access method further includes the following steps:

step 011: and acquiring the kerberos security authentication files corresponding to the distributed database clusters respectively.

Step 012: and respectively extracting authentication configuration information corresponding to each distributed database cluster from each kerberos security authentication file.

Step 013: and storing the one-to-one correspondence between the authentication configuration information and the cluster identification of each distributed database cluster into a combined authentication file.

Step 014: and respectively acquiring configuration files corresponding to the distributed database clusters based on the authentication configuration information in the combined authentication file.

Step 015: and respectively establishing long connections with the distributed database clusters according to the configuration files corresponding to the distributed database clusters, wherein each long connection corresponds to one connection thread.

Step 016: and updating the authentication ticket TGT which is used for carrying out the kerberos security authentication and corresponds to each distributed database cluster at regular time.

It will be appreciated that the authentication ticket TGTTGT is used to prove to the KDC service on the domain controller that the user has been authenticated by other domain controllers. The TGT is encrypted by the KRBTGT cryptographic hash and can be decrypted by any KDC service in the domain.

As can be seen from the above description, the distributed database cluster access method provided in the embodiment of the present application can further implement access to multiple clusters at the same time, can ensure effectiveness of cross-cluster access to multiple distributed database clusters by a user, and can effectively improve reliability of cross-cluster access to the distributed database clusters by the user.

In order to provide a specific process of user identity authentication to further improve the security of a user performing cross-cluster access to a distributed database cluster, in an embodiment of the distributed database cluster access method provided by the present application, referring to fig. 3, before step 100 of the distributed database cluster access method, the following is further included:

step 021: and generating a symmetric key by applying a symmetric encryption algorithm, and sending the symmetric key to a preset configuration center.

Step 022: and generating key objects of the asymmetric private keys and the public keys corresponding to the users by applying an asymmetric encryption algorithm, and encrypting the key objects by applying the symmetric encryption algorithm.

Step 023: storing each encrypted key object into a relational database, so that the configuration center obtains the encrypted key object corresponding to the target user from the relational database after receiving a registration request sent by each user, and sends a public key plaintext, an encrypted symmetric key and an encrypted private key object which respectively correspond to each key object to each corresponding user, so that each user respectively decrypts the encrypted symmetric key based on the obtained public key plaintext and decrypts the private key object based on the obtained symmetric key, and correspondingly obtains a private key for identity authentication.

It is understood that a specific example of the relational database may be a MySQL database, which is used to store configuration information such as authorization information and cluster information. The configuration center is used for managing key objects of all tenants on the intermediate service layer.

As can be seen from the above description, the distributed database cluster access method provided in the embodiment of the present application can further improve the security and timeliness of cross-cluster access performed by a user on a distributed database cluster, and the symmetric key is one-time pad, and the dynamic key is used to protect the private key, so that operation and maintenance personnel cannot obtain the private key of a tenant.

In order to provide a specific process of accessing an authorization query to further improve the security of a user performing cross-cluster access to a distributed database cluster, in an embodiment of the distributed database cluster access method provided in the present application, a user identifier of the target user is further included in the distributed database cluster access request, referring to fig. 4, where after step 100 and before step 200 of the distributed database cluster access method, the following contents are further specifically included:

step 031: and searching authorized user identifications corresponding to the cluster identifications of the target distributed database clusters in an authority control table arranged in a relational database.

Step 032: and judging whether the authorized user identification corresponding to the cluster identification of each target distributed database cluster comprises the user identification of the target user, if so, judging that each target distributed database cluster is a cross-user cluster which is authorized to be accessed by the target user.

As can be seen from the above description, the distributed database cluster access method provided in the embodiment of the present application can effectively improve reliability and efficiency of querying whether a current user has permission to access a cluster, and can further improve security and timeliness of cross-cluster access performed by the user on the distributed database cluster.

In terms of software, in order to solve the problem that an existing access method cannot simultaneously satisfy security, timeliness and effectiveness of cross-cluster access by a user for a distributed database cluster, the present application provides an embodiment of an intermediate service layer for implementing all or part of contents in the distributed database cluster access method, where the intermediate service layer may be a server or a server cluster, and in a specific example, the intermediate service layer may be composed of a plurality of application processes, see fig. 5, and the intermediate service layer specifically includes the following contents:

the request receiving module 10 is configured to obtain a distributed database cluster access request sent by a target user who has passed identity authentication, where the distributed database cluster access request includes cluster identifiers of a plurality of target distributed database clusters.

And a thread calling module 20, configured to, if it is determined that each target distributed database cluster is a cross-user cluster authorized to be accessed by the target user, locally search for a connection thread corresponding to each target distributed database cluster according to a cluster identifier of each target distributed database cluster, where each connection thread is created in advance based on a configuration file for kerberos security authentication and corresponding to each distributed database cluster, each configuration file is obtained in advance based on a merged authentication file stored locally, and the merged authentication file is used to store a one-to-one correspondence relationship between the cluster identifier of each distributed database cluster and authentication configuration information.

And the cluster access module 30 is configured to implement, based on the connection thread corresponding to each target distributed database cluster, access of the target user to each target distributed database cluster.

As can be seen from the above description, the intermediate service layer provided in the embodiment of the present application can effectively improve the security and the timeliness of cross-cluster access performed by a user on a distributed database cluster, and can implement access to multiple clusters at the same time, and can ensure the effectiveness of cross-cluster access performed by the user on multiple distributed database clusters, and can effectively improve the reliability of cross-cluster access performed by the user on the distributed database cluster, and can effectively reduce the maintenance cost of cross-cluster access, and further improve the cluster access efficiency by storing a configuration file locally, thereby being capable of safely, inexpensively, quickly and effectively solving the problem that cross-cluster data cannot be shared, improving the reliability and the intelligent degree of operation of the distributed database cluster in an enterprise, and effectively improving the user experience of a cluster access user.

In order to store the configuration files that are not covered by each other locally in the intermediate service layer in advance, so as to ensure the effectiveness of cross-cluster access to multiple distributed database clusters by a user, in an embodiment of the intermediate service layer provided by the present application, referring to fig. 6, the intermediate service layer further specifically includes the following contents:

a thread creation module 01, the thread creation module 01 being configured to:

As can be seen from the above description, the intermediate service layer provided in the embodiment of the present application can further implement access to multiple clusters simultaneously, can ensure effectiveness of cross-cluster access to multiple distributed database clusters by a user, and can effectively improve reliability of cross-cluster access to the distributed database clusters by the user.

In order to provide a specific process of user identity authentication to further improve security of a user performing cross-cluster access to a distributed database cluster, in an embodiment of an intermediate service layer provided in the present application, referring to fig. 7, the intermediate service layer further includes the following contents:

an identity authentication module 02, the identity authentication module 02 being configured to perform the following:

As can be seen from the above description, the intermediate service layer provided in the embodiment of the present application can further improve security and timeliness of cross-cluster access to a distributed database cluster by a user, and the operation and maintenance staff cannot obtain a private key of a tenant by using a dynamic key to protect the private key with a symmetric key in one-time pad.

In order to provide a specific process of accessing an authorization query, so as to further improve security of a user performing cross-cluster access to a distributed database cluster, in an embodiment of an intermediate service layer provided in the present application, a user identifier of the target user is further included in the distributed database cluster access request, referring to fig. 8, where the intermediate service layer further specifically includes the following contents:

a right query module 03, wherein the right query module 03 is configured to execute the following:

As can be seen from the above description, the intermediate service layer provided in the embodiment of the present application can effectively improve reliability and efficiency of querying whether a current user has a right to access a cluster, and can further improve security and efficiency of cross-cluster access performed by the user on a distributed database cluster.

In order to further explain the scheme, the application example relates to the internet big data technology, provides a system for realizing HBase cross-cluster authorized access and a method for realizing HBase cross-cluster authorized access, solves the problem of cluster security authentication by introducing a kerberos security authentication mechanism, packages a middle service layer at the same time, overcomes the problem of authentication failure of a single process for accessing a plurality of HBase clusters in the middle service layer, provides an authentication and authorization mechanism of a plurality of tenants, maintains the connection for accessing a plurality of clusters, thereby providing a system and a method for realizing HBase cross-cluster authorized access, and safely and low-cost solves the problem of unavailable sharing of cross-cluster data.

In order to achieve the purpose of preventing the cluster data from being tampered by malicious users, Kerberos is used for security authentication on the HBase cluster. However, because the Kerberos authentication addresses of different HBase clusters are different, krb5 files are loaded as a memory variable when the HBase clusters are connected, so that each Client process can only access one HBase cluster by using the original krb5 files.

In the intermediate service layer, in order to quickly access each cluster, the intermediate service layer maintains HBase long connections to a plurality of clusters in advance, but if krb5 files of a plurality of clusters are loaded in sequence, configuration files are mutually overlaid, and after the authentication of the next cluster is successful, the connection of the previous cluster is failed. Therefore, as a result of research, in order to simultaneously access a plurality of clusters and prevent the configuration files of the clusters from overlapping each other, in the intermediate service layer provided in the present application, it is first necessary to merge authentication files related to kerberos such as krb5. conf. The authentication configurations (such as kdc addresses) with difference among the clusters are extracted, the configurations of the clusters are combined into a file through a certain organization rule, and then the file is loaded into a memory variable, so that the authentication addresses of the clusters can be obtained simultaneously in one process. Meanwhile, the authentication ticket TGT has an authentication validity period, so that the ticket needs to be refreshed before the TGT is invalid.

The application of each upper-layer tenant establishes uniform access connection to HBase through the intermediate service layer of the application example of the application, and the intermediate service layer can be routed to each cluster through different cluster connections to obtain access data by receiving user requests for accessing different clusters. Considering that this operational flow involves data transfer between different tenants and the intermediate service layer, the intermediate service layer provides an authentication and authorization mechanism. In order to reduce the key maintenance cost of each tenant application operation and maintenance personnel, the authentication key of the application example is generated in a manner that a configuration center uniformly issues the key and a dynamic key protects a private key.

Referring to fig. 9, the distributed database cluster access system specifically includes: the system comprises an initialization module 1, an intermediate service layer tenant authentication module 2, an authorization execution module 3, a cluster connection creating and refreshing module 4 and a data access module 5.

The initialization module 1 is responsible for initialization work such as assigning tenants to applications, creating an authorization control table in a relational database, and the like. The intermediate service layer tenant authentication module 2 is responsible for verifying authentication and key distribution of each tenant on the intermediate service layer. The authorization execution module 3 is responsible for executing authorization operations of data tables between different applications. The cluster connection establishing and refreshing module 4 is responsible for ensuring that the connection is available, establishing the connection with each cluster through kerberos authentication and executing the refreshing operation before the bill expires. The data access module 5 is responsible for providing data access capabilities, e.g. so that tenant B can access tenant a's data across the cluster using its own application.

Referring to fig. 10, the initialization step executed by the initialization module 1 is as follows:

step 201: and (3) allocating tenants for the applications: an initialization module 1 in the intermediate service layer allocates tenants for each application. Assume that the tenant corresponding to the upstream application is tenant a, and the tenant corresponding to the downstream application is tenant B.

Step 202: and (4) creating an authority control table: a new authority control table is established in the relational database and used for recording the authorization information between applications, for example: the table name hbase _ acl _ info.

The fields include: application name app _ name, authorized table name table _ name, authorized application authorized _ app _ name, and authorized operations, etc.

Step 203: preparing a cluster configuration and kerberos security authentication related configuration file: assuming that data of an application of tenant a is stored on HBase cluster a1, and data of an application of tenant B is stored on HBase cluster B1, it is necessary to prepare related configuration information of HBase cluster a1 and HBase cluster B1, such as HBase-site.xml, in advance, and prepare configuration files (krb5.conf) and the like that cluster user userA and cluster user userB are respectively related to security authentication.

Step 204: and starting a main service process deployed on each server, establishing connection, and caching the authorization table in a memory. The authorization table information in the relational database is cached in the memory so as to improve the access efficiency.

(iii) each tenant accesses the HBase cluster through the connection provided by the intermediate service layer, and in order to prevent the data of each tenant from being modified by other tenants, the intermediate service layer provides an authentication and authorization mechanism, see fig. 11, and the procedure for the intermediate service layer tenant authentication module 2 to propagate the private key of each tenant is as follows:

step 301: generating a symmetric algorithm key: after the main service process is started, the intermediate service layer generates a symmetric algorithm key through a symmetric encryption algorithm (such as TripleDES) and stores the key in a file in an object serialization way, and then transmits the key file to the configuration center. After the main service process is restarted each time, the key is regenerated to achieve the effect of one-time pad.

Step 302: generating an asymmetric algorithm private key and public key object: the intermediate service layer generates an asymmetric algorithm private key and a public key object for each tenant by using an asymmetric encryption algorithm, the objects are encrypted by using a symmetric algorithm key and then stored in a relational database, and each tenant performs identity authentication by using the asymmetric key.

Step 303: initiating a user request and pulling a key object: and the configuration center sends a request, pulls the key objects of all tenants from the relational database, decrypts by using a symmetric algorithm and stores.

Step 304: initiating a registration request: the tenant application server initiates a registration request to the configuration center, the configuration management center automatically issues a new key according to the registration information, and the key file is stored by using the configuration center, which has the advantages that: all configurations are registered in the configuration center, and once the applications are omitted, the errors can be reported and can be timely discovered.

Step 305: distributing the key: the configuration center distributes the plain text of the public key of each tenant, the symmetric key of the public key encrypted by using a symmetric algorithm, each private key object encrypted by the symmetric key and three fields to all servers of each tenant.

Step 306: and realizing identity authentication: after receiving the information, each tenant application server decrypts the information by using the public key to obtain a symmetric key, and then decrypts the information by using the symmetric key to obtain a private key, so that the identity authentication is realized to access the intermediate service layer.

The symmetric algorithm has the advantages that: the encryption speed is high, the method is suitable for communication transmission data, and the defects are that the contents of the keys used by the encryption parties are consistent, and one party can be cracked after the leakage. The asymmetric key has the advantages that: private key encryption and public key decryption, the public key can be public, the contents of the public key and the private key are different, and the private key cannot be restored only by revealing the public key. Therefore, the private key is used for identity authentication, and the defects are as follows: the encryption speed is low, and the method is suitable for encrypting a small amount of information. According to the scheme, the characteristics of a big data cluster are combined, the symmetric key is one-time-pad, and the dynamic key is used for protecting the private key, so that operation and maintenance personnel cannot obtain the private key of a tenant.

And (IV) the authorization flow executed by the authorization execution module is as follows:

the tenant A authorizes the table of the tenant A to be accessed to the tenant B through the grantSelect authorization interface by providing the following authorization information, and the service process of the middle service layer writes the authorization information into the permission control table hbase _ acl _ info as shown in the table 1.

TABLE 1

Data write-in side	Data reader	Table name	Access mode
				Tenant A	Tenant B	A_TABLE_1	R (read only)
Tenant A	Tenant B	A_TABLE_2	R (read only)

Referring to fig. 12, after the initialization module 1 and the authorization execution module 3 are executed, a specific process of the distributed database cluster access method implemented by the application example of the present application is as follows:

step 401: requesting: the application server of the upper layer sends a request to an application process in the intermediate service layer.

Step 402: access authorization table: and the application process acquires the authorization information of the MySQL service from the authority control table of the relational database according to the application name and the table name of the request, and judges whether the access authority exists according to a certain rule.

Step 403: refreshing: if the access permission of the HBase exists, the cluster connection establishing and refreshing module 4 is used for establishing the safe connection with the HBase cluster A1 and the HBase cluster B1, and refreshing the connected bills at regular time.

Step 404: as a result: the specified connection is obtained and data access is provided by the data access module 5.

Referring to fig. 13, after the application server is started, the process of executing the right access rule in step 402 is as follows:

step 501: the upper layer application sends a request with application name a' and TABLE name a _ TABLE _1 to be accessed.

Step 502: whether application A' is the owner of the A _ TABLE _1 TABLE: and the main service process on each server receives the user request, and the main process adds a checkReadAccess () method according to the application name and the table name of the request to judge the authority. For example, the application name received in the request is a ', the TABLE name to be accessed is a _ TABLE _1, and it is determined whether the application a' is owner of the requested TABLE a _ TABLE _1, if so, step 503 is executed; if not, step 504 is performed.

Step 503: returning to the connection thread of the corresponding cluster of the application A', responding to the request: the application is the owner of the table, and the main thread should return the connection thread of the cluster corresponding to the application, and can respond to the request and return the result to the application.

Step 504: access right control TABLE, whether application a' has access right to a _ TABLE _1 TABLE: if the application is not the owner of the table, the access right control table hbase _ acl _ info determines if the application has access right to the table. If the authorization record exists in the authority control table in the table, executing step 505; otherwise, step 506 is performed.

Step 505: returning to the connection thread of the application corresponding to the cluster to which the A _ TABLE _1 TABLE belongs, responding to the request: the application is not the owner of the TABLE, but is authorized to have access to the TABLE, and the main thread should return the connection thread of the application corresponding to the cluster to which a _ TABLE _1 belongs, and can respond to the request and return the result to the application.

Step 506: throw "403-rights not enough" exception: application A' is not the table owner and is not authorized, has no rights to access this table, and throws a "403-rights insufficient" exception.

(seventhly) referring to fig. 14, the flow of creating and maintaining threads of the cluster connection creation and refresh module 4 is as follows, wherein long connections accessing different clusters are maintained in the main service process, each connection being created and maintained by a separate thread.

Step 601: and merging krb5 configuration files of each cluster: the krb5.conf files of the respective clusters prepared in the initialization module 1 are merged so that a plurality of HBase clusters can be accessed within a single process.

Step 602: sequentially loading HBase configuration files of each cluster, acquiring an access bill TGT of the cluster, and establishing long connection with the cluster: and sequentially loading the HBase configuration files of each cluster according to the sequence, acquiring an access bill TGT of each cluster, and establishing long connection with the clusters. For example, loading a configuration file of the HBase cluster A1, acquiring a bill from the user A to the HBase cluster A1, and creating a long connection to the HBase cluster A1; and loading the configuration file of the HBase cluster B1, acquiring a bill from the user B to the HBase cluster B1, creating a long connection to the HBase cluster B1 and the like.

Step 603: refreshing the bill operation before the bill expires: several hours before the ticket for each cluster expires, a refresh ticket operation is performed.

As can be seen from the above description, the distributed database cluster access system and method provided in the embodiments of the present application solve the problem of cluster security authentication through a kerberos security authentication mechanism, encapsulate a layer of intermediate service layer, overcome the problem of authentication failure of accessing multiple HBase clusters by a single process in the intermediate service layer, provide an authentication and authorization mechanism for multiple tenants, maintain connections for accessing multiple clusters, enable users of different clusters to access tables and data in other hbases clusters across clusters, and solve the problem of across-cluster data sharing with a secure, general, and low-cost method.

In order to solve the problem that an existing distributed database cluster access method cannot simultaneously satisfy security, timeliness and effectiveness of cross-cluster access of a user to a distributed database cluster from a hardware level, the present application provides an embodiment of an electronic device for implementing all or part of contents in the distributed database cluster access method, where the electronic device specifically includes the following contents:

fig. 15 is a schematic block diagram of a system configuration of an electronic device 9600 according to an embodiment of the present application. As shown in fig. 15, the electronic device 9600 can include a central processor 9100 and a memory 9140; the memory 9140 is coupled to the central processor 9100. Notably, this fig. 15 is exemplary; other types of structures may also be used in addition to or in place of the structure to implement telecommunications or other functions.

In one embodiment, the distributed database cluster access function may be integrated into a central processor. Wherein the central processor may be configured to control:

As can be seen from the above description, the electronic device provided in the embodiment of the present application can effectively improve the security and timeliness of cross-cluster access performed by a user on a distributed database cluster, and can implement access to multiple clusters at the same time, and can ensure the effectiveness of cross-cluster access performed by the user on multiple distributed database clusters, and can effectively improve the reliability of cross-cluster access performed by the user on the distributed database cluster, and can effectively reduce the maintenance cost of cross-cluster access, and further improve the cluster access efficiency by storing a configuration file locally, so that the problem that cross-cluster data cannot be shared can be safely, inexpensively, quickly, and effectively solved, the reliability and the intelligent degree of operation of the distributed database cluster in an enterprise can be improved, and the user experience of the cluster access user can be effectively improved.

In another embodiment, the intermediate service layer may be configured separately from the central processor 9100, for example, the intermediate service layer may be configured as a chip connected to the central processor 9100, and the distributed database cluster access function is realized by the control of the central processor.

As shown in fig. 15, the electronic device 9600 may further include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It is noted that the electronic device 9600 also does not necessarily include all of the components shown in fig. 15; further, the electronic device 9600 may further include components not shown in fig. 15, which can be referred to in the related art.

As shown in fig. 15, a central processor 9100, sometimes referred to as a controller or operational control, can include a microprocessor or other processor device and/or logic device, which central processor 9100 receives input and controls the operation of the various components of the electronic device 9600.

The memory 9140 can be, for example, one or more of a buffer, a flash memory, a hard drive, a removable media, a volatile memory, a non-volatile memory, or other suitable device. The information relating to the failure may be stored, and a program for executing the information may be stored. And the central processing unit 9100 can execute the program stored in the memory 9140 to realize information storage or processing, or the like.

The input unit 9120 provides input to the central processor 9100. The input unit 9120 is, for example, a key or a touch input device. Power supply 9170 is used to provide power to electronic device 9600. The display 9160 is used for displaying display objects such as images and characters. The display may be, for example, an LCD display, but is not limited thereto.

The memory 9140 can be a solid state memory, e.g., Read Only Memory (ROM), Random Access Memory (RAM), a SIM card, or the like. There may also be a memory that holds information even when power is off, can be selectively erased, and is provided with more data, an example of which is sometimes called an EPROM or the like. The memory 9140 could also be some other type of device. Memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application/function storage portion 9142, the application/function storage portion 9142 being used for storing application programs and function programs or for executing a flow of operations of the electronic device 9600 by the central processor 9100.

The memory 9140 can also include a data store 9143, the data store 9143 being used to store data, such as contacts, digital data, pictures, sounds, and/or any other data used by an electronic device. The driver storage portion 9144 of the memory 9140 may include various drivers for the electronic device for communication functions and/or for performing other functions of the electronic device (e.g., messaging applications, contact book applications, etc.).

The communication module 9110 is a transmitter/receiver 9110 that transmits and receives signals via an antenna 9111. The communication module (transmitter/receiver) 9110 is coupled to the central processor 9100 to provide input signals and receive output signals, which may be the same as in the case of a conventional mobile communication terminal.

Based on different communication technologies, a plurality of communication modules 9110, such as a cellular network module, a bluetooth module, and/or a wireless local area network module, may be provided in the same electronic device. The communication module (transmitter/receiver) 9110 is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130 to provide audio output via the speaker 9131 and receive audio input from the microphone 9132, thereby implementing ordinary telecommunications functions. The audio processor 9130 may include any suitable buffers, decoders, amplifiers and so forth. In addition, the audio processor 9130 is also coupled to the central processor 9100, thereby enabling recording locally through the microphone 9132 and enabling locally stored sounds to be played through the speaker 9131.

Embodiments of the present application further provide a computer-readable storage medium capable of implementing all the steps in the distributed database cluster access method in the foregoing embodiments, where the computer-readable storage medium stores thereon a computer program, and when the computer program is executed by a processor, the computer program implements all the steps of the distributed database cluster access method in the foregoing embodiments, where the execution subject is a server or a client, for example, when the processor executes the computer program, the processor implements the following steps:

From the above description, it can be seen that the computer-readable storage medium provided in the embodiments of the present application can effectively improve the security and the timeliness of the cross-cluster access of the user to the distributed database cluster, and can realize the access aiming at a plurality of clusters simultaneously, can ensure the effectiveness of cross-cluster access of a user to a plurality of distributed database clusters, and can effectively improve the reliability of the cross-cluster access of the user to the distributed database clusters, meanwhile, the maintenance cost of cross-cluster access can be effectively reduced, the cluster access efficiency is further improved by storing the configuration file locally, therefore, the problem that cross-cluster data cannot be shared can be safely, quickly and effectively solved at low cost, the reliability and the intelligent degree of operation of the distributed database cluster in an enterprise are improved, and the user experience of a cluster access user can be effectively improved.

As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, apparatus, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (devices), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

The principle and the implementation mode of the invention are explained by applying specific embodiments in the invention, and the description of the embodiments is only used for helping to understand the method and the core idea of the invention; meanwhile, for a person skilled in the art, according to the idea of the present invention, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present invention.

Claims

1. A distributed database cluster access method is characterized by comprising the following steps:

2. The method according to claim 1, further comprising, before the obtaining the request for accessing the distributed database cluster sent by the target user who has passed the identity authentication:

3. The method according to claim 1, further comprising, before the obtaining the request for accessing the distributed database cluster sent by the target user who has passed the identity authentication:

4. The method according to claim 1, wherein the request for access to the distributed database cluster further includes a user identifier of the target user;

5. An intermediary service layer, comprising:

6. The intermediate service layer as recited in claim 5, further comprising: a thread creation module to perform the following:

7. The intermediate service layer as recited in claim 5, further comprising: an identity authentication module for performing the following:

8. The intermediate service layer as claimed in claim 5, wherein the request for access to the distributed database cluster further includes a user identifier of the target user;

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the distributed database cluster access method of any one of claims 1 to 4 when executing the program.

10. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the distributed database cluster access method of any one of claims 1 to 4.