CN114490792A - Data processing method and device - Google Patents

Abstract

The present application relates to the field of database technologies, and in particular, to a data processing method and apparatus. In the processing method, a server receives a resource processing request sent by an application demander, the resource processing request carries a user name keyword, a role name keyword, a tenant name keyword, a HIVE library name keyword and an entitlement information keyword, the server determines an entity table of the user name, an entity table of the tenant name, an entity table of the role name and an entity table of the HIVE library name according to the keywords, and determines and stores at least one of a user role SQL statement, a tenant role relationship table statement and a role relationship table SQL statement according to the entity table of the user name, the entity table of the tenant name, the entity table of the role name, the entity table of the HIVE library name and the entitlement information keyword. The data processing method provided by the application can improve the accuracy of converting the required content sent by the required application party into the SQL statement.

Description

Data processing method and device

Technical Field

The present application relates to the field of database technologies, and in particular, to a data processing method and apparatus.

Background

With the increasing abundance of machine learning platform integration algorithms and big data processing modes, common requirements of big data platforms include the establishment of new users, tenants, roles, data warehouse tool (HIVE) libraries and corresponding empowerment in big data distributed computing (Hadoop) clusters.

In the prior art, when a big data Hadoop cluster newly establishes a user, a tenant, a role, a HIVE library and corresponding entitlements, a server receives demand content sent by a demand application party and displays the demand content, so that keyword information in the demand content is manually extracted, for example, the server receives a demand mail sent by the demand application party and displays mail content, and a technician extracts keyword information such as a user name, a tenant name, a role name, a HIVE library name and the entitlements from the demand mail; then the technical staff inputs the keyword information to the database, operates the database, searches an entity table corresponding to the keyword, the database sends the entity table to the server, the server displays the entity table, the technical staff generates a relational table Structured Query Language (SQL) statement according to the obtained entity table and the weighting information, then the SQL statement is input to the server, the server stores the SQL statement into a relational database management system (Oracle) table, and finally, a computer programming language (Java) code reads the SQL information in the Oracle to execute operations such as creating, weighting and directory capacity expansion of cluster users and roles.

However, in the process of using the method, it is found that the SQL statements corresponding to the required content sent by the required application party and finally stored in the database are prone to error, that is, the accuracy is low.

Therefore, how to improve the accuracy of converting the required content sent by the required application party into the SQL statement becomes an urgent technical problem to be solved.

Disclosure of Invention

The data processing method and device are used for improving the accuracy of converting the required content sent by the required application side into the SQL information.

In a first aspect, the present application provides a data processing method, applied to a server, including: receiving a resource processing request sent by an application demander, wherein the resource processing request carries a keyword of a user name, a keyword of a role name, a keyword of a tenant name, a keyword of a HIVE library name of a data warehouse tool and a keyword of entitlement information, and the keyword of the entitlement information is used for indicating the authority which is requested to be endowed by the application demander; determining an entity table of the user name, an entity table of the role name, an entity table of the tenant name and an entity table of the HIVE library name according to the resource processing request, wherein the entity table of the user name comprises an ID (identity) of the user name, the entity table of the role name comprises an ID of the role name, the entity table of the tenant name comprises an ID of the tenant name, and the entity table of the HIVE library name comprises an ID of the HIVE library name; determining at least one of a Structured Query Language (SQL) statement of a user role relationship table, an SQL statement of a tenant role relationship table and an SQL statement of a role HIVE relationship table according to the entity table of the user name, the entity table of the role name, the entity table of the tenant name, the entity table of the HIVE library name and the keywords of the empowerment information; and storing at least one of the user role relational table SQL statement, the tenant role relational table SQL statement and the role HIVE relational table SQL statement.

In the method, after receiving a resource processing request, a server directly obtains each keyword in the resource processing request, searches an ID corresponding to each keyword in a corresponding database based on the keyword, and establishes a relational table SQL statement together with every two IDs and entitlement information based on the IDs corresponding to each keyword. Compared with the prior art, the method has the advantages that the server replaces manual work to complete the operation of determining the corresponding ID based on the keywords, and the calculation efficiency and accuracy of the server are generally higher than those of manual work, so that the accuracy and efficiency of converting the required content sent by the required application party into the SQL statement of the relational table can be improved.

With reference to the first aspect, in a possible implementation manner, the determining, according to the resource processing request, an entity table of the user name, an entity table of the role name, an entity table of the tenant name, and an entity table of the HIVE library name includes: inquiring whether the user name, the role name, the tenant name and the HIVE library name exist in a local database according to the key words of the user name, the key words of the role name, the key words of the tenant name and the key words of the HIVE library name carried in the resource processing request; under the condition that the user name, the role name, the tenant name and the HIVE library name exist, acquiring the ID of the user name, the ID of the role name, the ID of the tenant name and the ID of the HIVE library name from the local database; determining an entity table of the user name according to the ID of the user name; determining an entity table of the role name according to the ID of the role name; determining a real-entity table of the tenant name according to the ID of the tenant name; and determining the entity table of the HIVE library name according to the ID of the HIVE library name.

In the implementation mode, whether the user name, the tenant name, the role name and the HIVE library name exist or not is determined according to the keyword information, under the existence condition, the ID corresponding to the corresponding keyword is determined in each entity table, and then the corresponding entity table is determined, so that the follow-up steps can be correctly executed, the follow-up condition which cannot be processed is avoided, and the success rate and the efficiency of converting the required content sent by the required application party into the SQL statement of the relation table are improved.

With reference to the first aspect, in a possible implementation manner, the method further includes: under the condition that at least one name of the user name, the role name, the tenant name and the HIVE library name does not exist in the local database, newly establishing an ID corresponding to the at least one name and the at least one name in the local database; and determining an entity table of the at least one name according to the newly-built at least one name and the ID corresponding to the at least one name.

In the implementation mode, if at least one name of the user name, the role name, the tenant name and the HIVE library name does not exist in the local database, that is, the name is a new name, an ID corresponding to the name and the corresponding name needs to be newly established in the local database, so that the follow-up steps can be correctly executed, processing failure is avoided, the success rate of converting the required content sent by the required application party into an SQL statement is improved, and the user requirement is met.

With reference to the first aspect, in a possible implementation manner, the determining, according to the entity table of the user name, the entity table of the role name, the entity table of the tenant name, the entity table of the HIVE library name, and the key of the weighting information, at least one of a user role relationship table Structured Query Language (SQL) statement, a tenant role relationship table (SQL) statement, and a role HIVE relationship table (SQL) statement includes: determining the SQL statement of the user role relation table according to the entity table of the user name, the entity table of the role name and the keywords of the weighting information; determining the SQL statement of the tenant role relationship table according to the entity table of the tenant name, the entity table of the role name and the keywords of the empowerment information; and determining the SQL statement of the role HIVE relational table according to the entity table of the role name, the entity table of the HIVE library name and the keywords of the empowerment information.

With reference to the first aspect, in a possible implementation manner, before determining, according to the resource processing request, the entity table of the username, the entity table of the role name, the entity table of the tenant name, and the entity table of the HIVE library name, the method further includes: and storing the keywords of the user name, the keywords of the role name, the keywords of the tenant name and the keywords of the HIVE library name.

In the implementation mode, the user name keyword, the role name keyword, the tenant name keyword, the HIVE library name keyword and other required information which are filled by the application demander are stored, and then the user name keyword, the tenant name keyword, the HIVE library name keyword and other required information can be used as account opening information, so that technicians can conveniently check or perform other processing in the future.

With reference to the first aspect, in a possible implementation manner, the storing at least one of the user role relationship table SQL statement, the tenant role relationship table SQL statement, and the role HIVE relationship table SQL statement includes: and storing at least one of the user role relational table SQL statement, the tenant role relational table SQL statement and the role HIVE relational table SQL statement into the local database.

In the implementation mode, the obtained SQL statement of the relational table is definitely stored in a local database, and a foundation is provided for subsequent operations such as corresponding expansion, new construction and the like.

In a second aspect, the present application provides an apparatus for processing data. The processing means may comprise functional modules for implementing the method of the first aspect or any one of its possible implementations. The technical effects of the processing apparatus can refer to the technical effects brought by the corresponding method, which are not described herein again.

The processing device may include: the system comprises a receiving module, a resource processing module and a processing module, wherein the receiving module is used for receiving a resource processing request sent by an application demander, the resource processing request carries a user name keyword, a role name keyword, a tenant name keyword, a data warehouse tool HIVE library name keyword and an entitlement information keyword, and the entitlement information keyword is used for indicating the authority which is requested to be given by the application demander; a determining module, configured to determine, according to the resource processing request, an entity table of the user name, an entity table of the role name, an entity table of the tenant name, and an entity table of the HIVE library name, where the entity table of the user name includes an ID of the user name, the entity table of the role name includes an ID of the role name, the entity table of the tenant name includes an ID of the tenant name, and the entity table of the HIVE library name includes an ID of the HIVE library name; the determining module is further configured to determine at least one of a user role relationship table Structured Query Language (SQL) statement, a tenant role relationship table (SQL) statement and a role HIVE relationship table (SQL) statement according to the entity table of the user name, the entity table of the role name, the entity table of the tenant name, the entity table of the HIVE library name and the keywords of the entitlement information; and the storage module is used for storing at least one of the user role relational table SQL statement, the tenant role relational table SQL statement and the role HIVE relational table SQL statement.

With reference to the second aspect, in a possible implementation manner, the determining module is specifically configured to: inquiring whether the user name, the role name, the tenant name and the HIVE library name exist in a local database according to the key words of the user name, the key words of the role name, the key words of the tenant name and the key words of the HIVE library name carried in the resource processing request; under the condition that the user name, the role name, the tenant name and the HIVE library name exist, acquiring the ID of the user name, the ID of the role name, the ID of the tenant name and the ID of the HIVE library name from the local database; determining an entity table of the user name according to the ID of the user name; determining an entity table of the role name according to the ID of the role name; determining a real-entity table of the tenant name according to the ID of the tenant name; and determining the entity table of the HIVE library name according to the ID of the HIVE library name.

With reference to the second aspect, in a possible implementation manner, the apparatus further includes: a new building module, configured to build, in the local database, at least one name of the user name, the role name, the tenant name, and the HIVE library name and an ID corresponding to the at least one name, when the at least one name does not exist in the local database; the determining module is further configured to determine an entity table of the at least one name according to the newly created at least one name and the ID corresponding to the at least one name.

With reference to the second aspect, in a possible implementation manner, the determining module is further configured to: determining the SQL statement of the user role relation table according to the entity table of the user name, the entity table of the role name and the keywords of the weighting information; determining the SQL statement of the tenant role relationship table according to the entity table of the tenant name, the entity table of the role name and the keywords of the empowerment information; and determining the SQL statement of the role HIVE relational table according to the entity table of the role name, the entity table of the HIVE library name and the keywords of the empowerment information.

With reference to the second aspect, in a possible implementation manner, the storage module is further configured to store the keyword of the user name, the keyword of the role name, the keyword of the tenant name, and the keyword of the HIVE library name before determining the entity table of the user name, the entity table of the role name, the entity table of the tenant name, and the entity table of the HIVE library name according to the resource processing request.

With reference to the second aspect, in a possible implementation manner, the storage module is specifically configured to: and storing at least one of the user role relational table SQL statement, the tenant role relational table SQL statement and the role HIVE relational table SQL statement into the local database.

With reference to the second aspect, in a possible implementation manner, the resource processing request further includes: the distributed file system HDFS directory capacity expansion requirement.

In a third aspect, the present application provides a server, comprising: a memory and a processor; the memory is to store program instructions; the processor is configured to invoke program instructions in the memory to perform a method according to the first aspect or any one of its possible implementations.

In a fourth aspect, the present application provides a computer-readable storage medium having stored thereon computer-executable instructions for implementing the method according to the first aspect or any one of the possible implementations thereof when executed by a processor.

In a fifth aspect, the present application provides a computer program product comprising a computer program which, when executed by a processor, is configured to implement the method according to the first aspect or any one of the possible implementations thereof.

In the data processing method provided by the application, a server receives a resource processing request sent by an application demand side, the resource processing request carries a user name keyword, a role name keyword, a tenant name keyword, a HIVE library name keyword and an entitlement information keyword, the server determines a user name entity table, a tenant name entity table, a role name entity table and an HIVE library name entity table according to the keywords, and then determines and stores at least one of a user role SQL statement, a tenant relation table SQL statement and a role HIVE relation table SQL statement according to the user name entity table, the tenant name entity table, the role name entity table, the HIVE library name entity table and the entitlement information keyword. Compared with the prior art, the data processing method has the advantages that the server replaces manual work to complete the operation of determining the corresponding ID based on the keywords, and the calculation efficiency and accuracy of the server are generally higher than those of manual work, so that the accuracy and efficiency of converting the required content sent by the required application party into the SQL statement of the relational table can be improved.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and together with the description, serve to explain the principles of the application.

FIG. 1 is a schematic diagram of a system architecture according to an embodiment of the present application;

FIG. 2 is a flow chart of a method for processing data according to an embodiment of the present application;

FIG. 3 is a block diagram of a data processing framework provided in accordance with an embodiment of the present application;

FIG. 4 is a schematic diagram of a Hadoop cluster resource application system according to an embodiment of the present application;

FIG. 5 is a schematic diagram of a data processing apparatus according to an embodiment of the present application;

fig. 6 is a schematic view of an apparatus according to another embodiment of the present application.

With the above figures, there are shown specific embodiments of the present application, which will be described in more detail below. These drawings and written description are not intended to limit the scope of the inventive concepts in any manner, but rather to illustrate the inventive concepts to those skilled in the art by reference to specific embodiments.

Detailed Description

Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

With the increasing abundance of machine learning platform integration algorithms and big data processing modes, it is a common requirement of a big data platform to create users, tenants, roles, a data warehouse tool (HIVE) library and corresponding empowerments in a big data distributed computing (Hadoop) cluster.

In the prior art, when a big data Hadoop cluster newly establishes a user, a tenant, a role, a HIVE library and corresponding entitlements, a server receives demand content sent by a demand application party and displays the demand content, so that keyword information in the demand content is manually extracted, for example, the server receives a demand mail sent by the demand application party and displays mail content, and a technician extracts keyword information such as a user name, a tenant name, a role name, a HIVE library name and the entitlements from the demand mail; then the technical personnel inputs the keyword information into the database, operates the database, searches or newly builds entity tables (user tables, tenant tables, role tables and HIVE tables) corresponding to user names, tenant names, role names and HIVE library names, the database sends the entity tables to the server, the server displays the entity tables, the technical personnel generates Structured Query Language (SQL) sentences of the relation tables related to every two main keys according to the obtained entity tables and the weighting information, then inputs the SQL sentences into the server, the server stores the SQL sentences into the Oracle table, and finally reads the SQL information in the Oracle by Java codes to execute operations of cluster users, role creation, weighting, directory capacity expansion and the like.

However, the method for extracting the keywords in the required content sent by the required application party manually and finally converting the required content into the SQL statement is inefficient, and there are many tables involved in the process, which makes human error easily.

In view of the above, the present application provides a data processing method, which aims to solve the above technical problems in the prior art.

The following describes the technical solutions of the present application and how to solve the above technical problems with specific examples. The following several specific embodiments may be combined with each other, and details of the same or similar concepts or processes may not be repeated in some embodiments. Embodiments of the present application will be described below with reference to the accompanying drawings.

Fig. 1 is a schematic diagram of a system architecture provided in an embodiment of the present application, where the system architecture includes a client 101 and a server 102. The connection between the client 101 and the server 102 may be direct or indirect through wired or wireless communication, and the application is not limited herein.

Optionally, when an application demander needs to perform operations such as creating a cluster user and a role, weighting, and expanding a directory, a resource processing request is sent to the server cluster through the client 101, and then the resource processing request is distributed to the corresponding server 102 by an Application Programming Interface (API) gateway, and then the server 102 processes the resource processing request to obtain the relational table SQL statement.

Further, the server 102 may store the SQL statement in an Oracle table, and finally, the Java code reads the SQL information in the Oracle to execute operations such as creating a cluster user and a role, assigning a right, and expanding a directory.

With reference to the system architecture shown in fig. 1, fig. 2 is a flowchart of a data processing method provided in an embodiment of the present application, and is applied to the server 102. As shown in fig. 2, the method provided in the embodiment of the present application includes S201, S202, S203, and S204. The individual steps in the method shown in fig. 2 are explained in detail below.

As an example, fig. 3 is a schematic diagram of a data processing framework provided in an embodiment of the present application. The method shown in fig. 2 is described in detail below in conjunction with fig. 3.

S201, receiving a resource processing request sent by an application demander, wherein the resource processing request carries a user name keyword, a role name keyword, a tenant name keyword, a data warehouse tool HIVE library name keyword and an entitlement information keyword, and the entitlement information keyword is used for indicating the authority which is requested to be given by the application demander.

In this step, the application demander may directly fill in a resource processing request on the client 101 according to information on the system web page, for example, input a keyword of a user name for opening an account, a keyword of a role name, a keyword of a tenant name, a keyword of a HIVE library name of a data warehouse tool, and a keyword of entitlement information, and then the server 102 receives the resource processing request.

Optionally, the resource processing request further includes: distributed File System (HDFS) directory capacity expansion requirements are that a directory needs to be expanded to a specified value.

Optionally, the resource processing request may carry a Uniform Resource Locator (URL) tag, where the URL tag information is used to indicate a keyword of a user name, a keyword of a role name, a keyword of a tenant name, a keyword of a data warehouse tool HIVE library name, and a keyword of entitlement information.

It should be noted that, there is a certain relationship among the above-mentioned users, tenants and roles, and specifically, a user is an API consuming entity with authentication information, and may belong to multiple tenants/projects/organizations. The tenants are resource sets which can be accessed by some services, the resource sets can be used by a plurality of users, the tenants are independent from each other, and the information of other tenants cannot be viewed under the current tenant. A role is a user's property that can perform a particular series of operations, and specifies a series of rights and privileges for a user in a certain tenant. Furthermore, the tenant may have a plurality of roles and users, and the user may have a plurality of roles and tenants, and the roles give the user authority to operate functions.

As an example, a company employee needs to apply for a business expense reimbursement from the company financial department, and a user represents employee A who holds relevant information, such as name, job number, email, etc.; the tenant corresponds to a project group, and the employee A can belong to different project groups at the same time; when the employee a makes a request for reimbursement of business charge, one of the project groups to which the employee a belongs must be specified, and the role specifies the authority which the employee a has in a certain project group, such as what charge can be reimbursed and what cannot be reimbursed.

Understandably, the HIVE library is a data warehouse tool based on Hadoop, is used for data extraction, transformation and loading, and is a mechanism for querying and analyzing large-scale data stored in Hadoop. HIVE can map a structured data file into a database table and provide SQL query functions.

Optionally, the resource processing request (including the user name keyword, the role name keyword, the tenant name keyword, the data warehouse tool HIVE library name keyword, and the entitlement information keyword) filled by the application demander on the client 101 may be directly stored in Oracle as an account opening information list, which is convenient for viewing in later arrangement.

Optionally, the user name, the role name, the tenant name and the HIVE library name may be automatically spliced according to the key of the user name, the key of the role name, the key of the tenant name and the key of the HIVE library name.

Illustratively, as shown in fig. 3, the application demander fills out the keyword information: the application name is XXX, the HIVE library capacity is 1T, the scheduling mode needing to be opened is A, and the keywords of the application name comprise a USER name 'USER _ XXX', a ROLE name 'ROLE _ XXX', a tenant name 'TANNET _ XXX' and a HIVE library name 'HIVE _ XXX'. And meanwhile, storing the filled keyword information and information of an applicant, an application department and the like into a database to be used as an account opening list.

S202, according to the resource processing request, determining an entity table of a user name, an entity table of a role name, an entity table of a tenant name and an entity table of an HIVE library name, wherein the entity table of the user name comprises an ID of the user name, the entity table of the role name comprises an ID of the role name, the entity table of the tenant name comprises an ID of the tenant name, and the entity table of the HIVE library name comprises an ID of the HIVE library name.

In the step, whether the user name, the role name, the tenant name and the HIVE library name exist or not is inquired in a local database according to the key words of the user name, the role name, the tenant name and the HIVE library name.

Optionally, if the user name, the role name, the tenant name and the HIVE library name exist in the local database, the ID of the user name, the ID of the role name, the ID of the tenant name and the ID of the HIVE library name are obtained from the local database. Further, according to the ID of the user name, the ID of the role name, the ID of the tenant name and the ID of the HIVE library name, the entity table of the user name, the entity table of the role name, the entity table of the tenant name and the entity table of the HIVE library name are determined.

Specifically, determining an entity table of the user name according to the ID of the user name; determining an entity table of the role name according to the ID of the role name; determining a real-entity table of the tenant name according to the ID of the tenant name; and determining an entity table of the HIVE library name according to the ID of the HIVE library name.

Optionally, if at least one of the user name, the role name, the tenant name and the HIVE library name does not exist in the local database, the at least one name and the ID corresponding to the at least one name are newly established in the local database. Further, according to the newly-created at least one name and the ID corresponding to the at least one name, determining an entity table of the at least one name. And adding one to the newly-built ID corresponding to the at least one name according to the maximum ID in the database.

Exemplarily, as shown in fig. 3, four entity tables are obtained, that is, a USER table, a ROLE table, a tenant table, and a save table, where the USER table includes information such as a USER NAME "USER _ NAME" and an ID "USER _ ID" corresponding to the USER NAME, the ROLE table includes information such as a ROLE NAME "roll _ NAME" and an ID "roll _ ID" corresponding to the ROLE NAME, the tenant table includes information such as a tenant NAME "TANNET _ NAME" and an ID "TANNET _ ID" corresponding to the tenant NAME, and the save table includes information such as a save library NAME "save _ NAME" and an ID "save _ ID" corresponding to the save library NAME.

In the embodiment, whether the user name, the tenant name, the role name and the HIVE library name exist in the local database or not is judged, and then the corresponding ID or the new name and the ID corresponding to the new name are searched, so that the follow-up steps can be correctly executed, the processing failure is avoided, the success rate of converting the required content sent by the required application party into the SQL statement is improved, and the user requirement is met.

S203, determining at least one of a user role relationship table SQL statement, a tenant role relationship table SQL statement and a role HIVE relationship table SQL statement according to the entity table of the user name, the entity table of the role name, the entity table of the tenant name, the entity table of the HIVE library name and the keywords of the empowerment information.

In this step, the relationship table is associated with the corresponding primary key ID in the entity table used for pairwise weighting.

Specifically, according to an entity table of a user name, an entity table of a role name and keywords of the weighting information, determining a user role relationship table SQL statement; determining a tenant role relationship table SQL statement according to an entity table of a tenant name, an entity table of a role name and keywords of the empowerment information; and determining the SQL statement of the role HIVE relational table according to the entity table of the role name, the entity table of the HIVE library name and the key words of the weighting information.

As an example, when a user XXX applies for a new CREATE (CREATE) authority of a HIVE library (HIVE _ XXX) entitled ROLE (ROLE _ XXX), a HIVE _ ID searched by the HIVE _ XXX, a ROLE _ ID searched by the ROLE _ XXX in a local database and a CTRATE generate a ROLE HIVE library relational table SQL statement.

Illustratively, as shown in fig. 3, at least one of a USER ROLE relationship table, a tenant ROLE relationship table, and a ROLE high relationship table, for example, the USER ROLE relationship table in the figure, is obtained, and the relationship table includes information such as ID "USER _ ID" corresponding to a USER name and ID "roll _ ID" corresponding to a ROLE name, and new entitlement information (CREATE).

S204, storing at least one of the user role relational table SQL statement, the tenant role relational table SQL statement and the role HIVE relational table SQL statement.

In the step, at least one of the obtained user role relational table SQL statement, tenant role relational table SQL statement and role HIVE relational table SQL statement is stored in Oracle, and further, data information in Oracle is read by Java codes and operations such as creating cluster users and roles, assigning rights and expanding directory are executed.

According to the method in the embodiment, the step of obtaining the keywords according to the application demand side mail is weakened originally, the opening keywords are filled in directly according to the information on the system webpage, and the system directly generates the relational table SQL sentences after filling in. The step of extracting the information keywords in the mail by the account opening personnel is reduced; and manual steps are reduced, the probability of errors caused by manual work is reduced, and the accuracy and efficiency of converting the required content sent by the required application party into the SQL statement can be improved.

Based on the foregoing embodiment, fig. 4 is a schematic diagram of a Hadoop cluster resource application system provided in an embodiment of the present application, where the system includes the server 102 in the foregoing embodiment. The Hadoop cluster resource application system comprises: a server (Nginx) cluster module 401, an API gateway module 402, a load balancing and fusing downgrade module 403, a registry and configuration center module 404, a link tracking and monitoring alarm module 405, a persistence module 406, a client (Feign) call module 407, and a sprinbot (SpringBoot) module 408, which may communicate with each other. The system is built based on the environment of a world wide WEB (WEB) system.

The Nginx cluster module 401 is configured to receive a resource processing request, distribute the resource processing request, and respond to the resource processing request. The API gateway module 402 is used to publish, manage, protect, and monitor APIs, and implement cross-system and cross-protocol service capability interworking. The load balancing and fuse downgrading module 403 is used to protect the resource when the resource applies for excessive system access in a short time. The registry and configuration center module 404 is responsible for managing the registration and configuration of the various services. The link tracking and monitoring alarm module 405 is used for tracking analysis and alarming when a problem occurs in the system. The persistent layer in the persistent module 406 includes an Oracle module, a remote dictionary service (Redis) module, and the Redis module is used for storing frequently used and infrequently changed data; the Feign module 407 is configured to provide a client for calling between services, and the SpringBoot module 408 is configured to generate an SQL statement.

Understandably, the output of the Nginx cluster module 401 is input into the API gateway module 402, the output of the API gateway module 402 is input into the load balancing and fusing downgrading module 403, the registry and the configuration center module 404, the output of the load balancing and fusing downgrading module 403, the registry and the configuration center module 404 is input into the SpringBoot module 408, and the output of the SpringBoot module 408 is input into the persistence module 406 and the link tracking and monitoring alarm module 405.

Specifically, the application demander inputs a resource processing request to the Nginx cluster module 401, the Nginx cluster module 401 transfers all the requests to the API gateway module 402, and the gateway dynamically routes the current request to a specific service (account opening, right giving, capacity expansion, or the like), thereby generating an SQL statement. The registry and configuration center module 404 may register the service to a registry micro service platform (Nacos) regarding the specified service, and at the same time, use the Nacos as a configuration center to facilitate changing the configuration file information at any time, and the service may use the Feign call module 407 to make mutual call (for example, in fig. 3, service 1 and service 4 may be called each other, service 2 and service 5 may be called each other, and service 3 and service 6 may be called each other). A load balancing and blowing degradation module 403 (e.g., a client load balancing tool (Ribbon)) may be used to invoke a service when the number of times a service is invoked is excessive, and the service may be blown and degraded at the gateway when a problem arises with the service (sentinel). Further, using a link tracking and monitoring alarm module 405 (for example, link tracking (Sleuth) and distributed tracking system (Zipkin) technology), information of each service is delivered to a service monitoring system and a time sequence database (Prometheus) for aggregation analysis, then a data visualization tool (Grafana) is used for visualization display, an alarm module (alteranger) provided by Prometheus is used for obtaining alarm information of the service in real time, and the alarm information is notified to development and maintenance personnel in a mail or mobile phone short message manner.

On the basis of the above embodiments, fig. 5 shows a device 500 for processing data provided by the embodiments of the present application, where the device 500 includes: a receiving module 501, a determining module 502, a storing module 503 and a new building module 504.

The receiving module 501 is configured to receive a resource processing request sent by an application demander, where the resource processing request carries a keyword of a user name, a keyword of a role name, a keyword of a tenant name, a keyword of a data warehouse tool HIVE library name, and a keyword of entitlement information, and the keyword of the entitlement information is used to indicate an authority requested to be granted by the application demander; a determining module 502, configured to determine, according to the resource processing request, an entity table of the user name, an entity table of the role name, an entity table of the tenant name, and an entity table of the HIVE library name, where the entity table of the user name includes an ID of the user name, the entity table of the role name includes an ID of the role name, the entity table of the tenant name includes an ID of the tenant name, and the entity table of the HIVE library name includes an ID of the HIVE library name; the determining module 502 is further configured to determine at least one of a user role relationship table Structured Query Language (SQL) statement, a tenant role relationship table (SQL) statement and a role HIVE relationship table (SQL statement) statement according to the entity table of the user name, the entity table of the role name, the entity table of the tenant name, the entity table of the HIVE library name and the keywords of the entitlement information; the storage module 503 is configured to store at least one of the user role relationship table SQL statement, the tenant role relationship table SQL statement, and the role HIVE relationship table SQL statement.

As an example, the apparatus 500 may be configured to perform the method shown in fig. 2, for example, the receiving module 501 is configured to perform S201, the determining module 502 is configured to perform S202 and S203, and the storing module 503 is configured to perform S204.

In a possible implementation manner, the determining module 502 is specifically configured to: inquiring whether the user name, the role name, the tenant name and the HIVE library name exist in a local database according to the key words of the user name, the role name, the tenant name and the HIVE library name carried in the resource processing request; under the condition that the user name, the role name, the tenant name and the HIVE library name exist, acquiring the ID of the user name, the ID of the role name, the ID of the tenant name and the ID of the HIVE library name from a local database; determining an entity table of the user name according to the ID of the user name; determining an entity table of the role name according to the ID of the role name; determining a real-entity table of the tenant name according to the ID of the tenant name; and determining an entity table of the HIVE library name according to the ID of the HIVE library name.

In one possible implementation, the apparatus further includes: a new building module 504, configured to build at least one name and an ID corresponding to the at least one name in the local database when the at least one name does not exist in the local database; the determining module 502 is further configured to determine an entity table of at least one name according to the newly created at least one name and the ID corresponding to the at least one name.

In a possible implementation manner, the determining module 502 is further configured to determine a user role relationship table SQL statement according to an entity table of a user name, an entity table of a role name, and a keyword of the weighting information; determining a tenant role relationship table SQL statement according to an entity table of a tenant name, an entity table of a role name and keywords of the empowerment information; and determining the SQL statement of the role HIVE relational table according to the entity table of the role name, the entity table of the HIVE library name and the key words of the weighting information.

In a possible implementation manner, the storage module 503 is further configured to store the keywords of the user name, the keywords of the role name, the keywords of the tenant name, and the keywords of the HIVE library name before determining the entity table of the user name, the entity table of the role name, the entity table of the tenant name, and the entity table of the HIVE library name according to the resource processing request.

In a possible implementation manner, the storage module 503 is specifically configured to: and storing at least one of the user role relational table SQL statement, the tenant role relational table SQL statement and the role HIVE relational table SQL statement in a local database.

Fig. 6 is a schematic view of an apparatus according to another embodiment of the present application. The apparatus shown in fig. 6 may be used to perform the method of any of the previous embodiments.

As shown in fig. 6, the apparatus 600 of the present embodiment includes: memory 601, processor 602, communication interface 603, and bus 604. The memory 601, the processor 602, and the communication interface 603 are communicatively connected to each other via a bus 604.

The memory 601 may be a Read Only Memory (ROM), a static memory device, a dynamic memory device, or a Random Access Memory (RAM). The memory 601 may store a program, and the processor 602 is configured to perform the steps of the method shown in the above-described embodiments when the program stored in the memory 601 is executed by the processor 602.

The processor 602 may be a general-purpose Central Processing Unit (CPU), a microprocessor, an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits, and is configured to execute related programs to implement the methods shown in the embodiments of the present application.

The processor 602 may also be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the method of the embodiment of the present application may be implemented by integrated logic circuits of hardware or instructions in the form of software in the processor 602.

The processor 602 may also be a general purpose processor, a Digital Signal Processor (DSP), an ASIC, an FPGA (field programmable gate array) or other programmable logic device, discrete gate or transistor logic device, or discrete hardware components. The various methods, steps and logic blocks disclosed in the embodiments of the present application may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

The steps of the method disclosed in connection with the embodiments of the present application may be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software module may be located in ram, flash memory, rom, prom, or eprom, registers, etc. storage media as is well known in the art. The storage medium is located in the memory 601, and the processor 602 reads the information in the memory 601, and in combination with hardware of the processor, performs functions required by units included in the apparatus of the present application.

The communication interface 603 may enable communication between the apparatus 600 and other devices or communication networks using, but not limited to, transceiver means.

Bus 604 may include a pathway to transfer information between various components of apparatus 600 (e.g., memory 601, processor 602, communication interface 603).

It should be understood that the apparatus 600 shown in the embodiment of the present application may be an electronic device, or may also be a chip configured in the electronic device.

The data processing method and apparatus in the present application may be used in the field of database technology, and may also be used in any field other than the field of database technology. The application of the data processing method and the data processing device is not limited in the application field.

It should be understood that the term "and/or" herein is merely one type of association relationship that describes an associated object, meaning that three relationships may exist, e.g., a and/or B may mean: a exists singly, A and B exist simultaneously, and B exists singly, wherein A and B can be singular or plural. In addition, the "/" in this document generally indicates that the former and latter associated objects are in an "or" relationship, but may also indicate an "and/or" relationship, and may be understood with particular reference to the former and latter contexts.

In this application, "at least one" means one or more, "a plurality" means two or more. "at least one of the following" or similar expressions refer to any combination of these items, including any combination of the singular or plural items. For example, at least one (one) of a, b, or c, may represent: a, b, c, a-b, a-c, b-c or a-b-c, wherein a, b and c can be single or multiple.

It should be understood that, in the various embodiments of the present application, the sequence numbers of the above-mentioned processes do not mean the execution sequence, and the execution sequence of each process should be determined by its function and inherent logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.

In the several embodiments provided in the present application, it should be understood that the disclosed system, apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, a division of a unit is merely a logical division, and an actual implementation may have another division, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.

Units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit.

The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application or portions thereof that substantially contribute to the prior art may be embodied in the form of a software product stored in a storage medium and including instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: u disk, removable hard disk, read only memory, random access memory, magnetic or optical disk, etc. for storing program codes.

The above description is only for the specific embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present application, and shall be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims (10)

1. A data processing method is applied to a server and comprises the following steps:

receiving a resource processing request sent by an application demander, wherein the resource processing request carries a keyword of a user name, a keyword of a role name, a keyword of a tenant name, a keyword of a HIVE library name of a data warehouse tool and a keyword of entitlement information, and the keyword of the entitlement information is used for indicating the authority which is requested to be endowed by the application demander;

determining an entity table of the user name, an entity table of the role name, an entity table of the tenant name and an entity table of the HIVE library name according to the resource processing request, wherein the entity table of the user name comprises an ID (identity) of the user name, the entity table of the role name comprises the ID of the role name, the entity table of the tenant name comprises the ID of the tenant name, and the entity table of the HIVE library name comprises the ID of the HIVE library name;

determining at least one of a user role relationship table Structured Query Language (SQL) statement, a tenant role relationship table (SQL) statement and a role HIVE relationship table (SQL) statement according to the entity table of the user name, the entity table of the role name, the entity table of the tenant name, the entity table of the HIVE library name and the keywords of the empowerment information;

and storing at least one of the user role relational table SQL statement, the tenant role relational table SQL statement and the role HIVE relational table SQL statement.

2. The method of claim 1, wherein determining the entity table for the username, the entity table for the role name, the entity table for the tenant name, and the entity table for the HIVE library name based on the resource handling request comprises:

inquiring whether the user name, the role name, the tenant name and the HIVE library name exist in a local database according to the key words of the user name, the key words of the role name, the key words of the tenant name and the key words of the HIVE library name carried in the resource processing request;

under the condition that the user name, the role name, the tenant name and the HIVE library name exist, acquiring the ID of the user name, the ID of the role name, the ID of the tenant name and the ID of the HIVE library name from the local database;

determining an entity table of the user name according to the ID of the user name;

determining an entity table of the role name according to the ID of the role name;

determining a real-entity table of the tenant name according to the ID of the tenant name;

and determining the entity table of the HIVE library name according to the ID of the HIVE library name.

3. The method of claim 2, further comprising:

under the condition that at least one name of the user name, the role name, the tenant name and the HIVE library name does not exist in the local database, newly establishing an ID corresponding to the at least one name and the at least one name in the local database;

and determining an entity table of the at least one name according to the newly-built at least one name and the ID corresponding to the at least one name.

4. The method of claim 1, wherein determining at least one of a user role relationship table Structured Query Language (SQL) statement, a tenant role relationship table (SQL) statement and a role HIVE relationship table (SQL) statement according to the entity table of the user name, the entity table of the role name, the entity table of the tenant name, the entity table of the HIVE library name and the key of the entitlement information comprises:

determining the SQL statement of the user role relation table according to the entity table of the user name, the entity table of the role name and the keywords of the empowerment information;

determining the SQL statement of the tenant role relationship table according to the entity table of the tenant name, the entity table of the role name and the keywords of the empowerment information;

and determining the SQL statement of the role HIVE relational table according to the entity table of the role name, the entity table of the HIVE library name and the keywords of the empowerment information.

5. The method of any of claims 1-4, wherein prior to determining the entity table for the username, the entity table for the role name, the entity table for the tenant name, and the entity table for the HIVE library name from the resource handling request, the method further comprises:

and storing the keywords of the user name, the keywords of the role name, the keywords of the tenant name and the keywords of the HIVE library name.

6. The method of any of claims 1-4, wherein storing at least one of the user role relationship table (SQL) statement, the tenant role relationship table (SQL) statement, and the role HIVE relationship table (SQL) statement comprises:

and storing at least one of the user role relational table SQL statement, the tenant role relational table SQL statement and the role HIVE relational table SQL statement into a local database.

7. A data processing device applied to a server comprises:

the system comprises a receiving module, a resource processing module and a processing module, wherein the receiving module is used for receiving a resource processing request sent by an application demander, the resource processing request carries a user name keyword, a role name keyword, a tenant name keyword, a data warehouse tool HIVE library name keyword and an entitlement information keyword, and the entitlement information keyword is used for indicating the authority which is requested to be given by the application demander;

a determining module, configured to determine, according to the resource processing request, an entity table of the user name, an entity table of the role name, an entity table of the tenant name, and an entity table of the HIVE library name, where the entity table of the user name includes an ID of the user name, the entity table of the role name includes an ID of the role name, the entity table of the tenant name includes an ID of the tenant name, and the entity table of the HIVE library name includes an ID of the HIVE library name;

the determining module is further configured to determine at least one of a user role relationship table Structured Query Language (SQL) statement, a tenant role relationship table (SQL) statement and a role HIVE relationship table (SQL) statement according to the entity table of the user name, the entity table of the role name, the entity table of the tenant name, the entity table of the HIVE library name and the keywords of the entitlement information;

and the storage module is used for storing at least one of the user role relational table SQL statement, the tenant role relational table SQL statement and the role HIVE relational table SQL statement.

8. A server, comprising: a memory and a processor;

the memory is to store program instructions;

the processor is configured to invoke program instructions in the memory to perform the method of any of claims 1 to 6.

9. A computer-readable storage medium having computer-executable instructions stored therein, which when executed by a processor, are configured to implement the method of any one of claims 1 to 6.

10. A computer program product, characterized in that a computer program is included in the computer program product, which computer program, when being executed by a processor, is adapted to carry out the method of any one of claims 1 to 6.

Images