CN113973017B - Business intelligent platform data processing system and method - Google Patents

Business intelligent platform data processing system and method Download PDF

Info

Publication number
CN113973017B
CN113973017B CN202111248721.0A CN202111248721A CN113973017B CN 113973017 B CN113973017 B CN 113973017B CN 202111248721 A CN202111248721 A CN 202111248721A CN 113973017 B CN113973017 B CN 113973017B
Authority
CN
China
Prior art keywords
data
accessed
platforms
business
business intelligent
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202111248721.0A
Other languages
Chinese (zh)
Other versions
CN113973017A (en
Inventor
朱辉
薛延波
张涛
赵鹏
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Huapin Borui Network Technology Co Ltd
Original Assignee
Beijing Huapin Borui Network Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Huapin Borui Network Technology Co Ltd filed Critical Beijing Huapin Borui Network Technology Co Ltd
Priority to CN202111248721.0A priority Critical patent/CN113973017B/en
Publication of CN113973017A publication Critical patent/CN113973017A/en
Application granted granted Critical
Publication of CN113973017B publication Critical patent/CN113973017B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L63/00—Network architectures or network communication protocols for network security
    • H04L63/08—Network architectures or network communication protocols for network security for authentication of entities
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90—Details of database functions independent of the retrieved data types
    • G06F16/95—Retrieval from the web
    • G06F16/955—Retrieval from the web using information identifiers, e.g. uniform resource locators [URL]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90—Details of database functions independent of the retrieved data types
    • G06F16/95—Retrieval from the web
    • G06F16/957—Browsing optimisation, e.g. caching or content distillation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L63/00—Network architectures or network communication protocols for network security
    • H04L63/04—Network architectures or network communication protocols for network security for providing a confidential data exchange among entities communicating through data packet networks
    • H04L63/0428—Network architectures or network communication protocols for network security for providing a confidential data exchange among entities communicating through data packet networks wherein the data content is protected, e.g. by encrypting or encapsulating the payload
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L63/00—Network architectures or network communication protocols for network security
    • H04L63/08—Network architectures or network communication protocols for network security for authentication of entities
    • H04L63/0807—Network architectures or network communication protocols for network security for authentication of entities using tickets, e.g. Kerberos
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/02—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes

Landscapes

  • Engineering & Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Databases & Information Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Security & Cryptography (AREA)
  • Computing Systems (AREA)
  • Computer Hardware Design (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Storage Device Security (AREA)

Abstract

The embodiment of the invention discloses a business intelligent platform data processing system, which comprises: the resource aggregation module is used for monitoring and aggregating the data of each business intelligent platform; the unified authority control module is used for managing the converged data based on the RBAC model; the data security module is used for performing login verification and access authority verification on each user when each user accesses data of menu resources and report boards through SDK integration or WEB; and the resource agent access module is used for performing secondary agent on the data accessed by each business intelligent platform based on Session. The embodiment of the invention also discloses a data processing method of the business intelligent platform. The invention solves the problems of disordered authorization and high data management cost, the problem of adaptation of the authority management of a plurality of business intelligent platforms and the problem of data security.

Description

Business intelligent platform data processing system and method
Technical Field
The invention relates to the technical field of computers, in particular to a business intelligent platform data processing system and a business intelligent platform data processing method.
Background
The business intelligence platform (BI platform) can greatly reduce the data acquisition cost, improve the data use efficiency, and can support business data analysis by using visualized and interactive operations. Among them, traditional BI platforms often focus on report development; the self-service BI platform is mainly oriented to business analysts, and performs self-service analysis of business leaders and emphasizes data analysis; there are also third party open source BI platforms, focusing on Internet analysis and mobile APP data analysis.
Therefore, in order to meet the business intelligence requirements in different scenes, a plurality of BI platforms need to be built. In the prior art, when a plurality of BI platforms manage the authority, the authority of each BI platform is independently deployed and managed, which causes confusion of authorization, causes exponential rise of data management cost, and also causes the problem of data security.
Disclosure of Invention
The invention aims to provide a business intelligent platform data processing system and a method, wherein a plurality of business intelligent platforms share a unified authority control module, so that the problems of disordered authorization and high data management cost are solved; by the design of plug-in data resource aggregation standardization and the simultaneous support of two resource synchronization modes of a Push mode and a Pull mode, the problem of permission management adaptation of a plurality of BI platforms is solved, and the data security problem is solved based on a triple protection mechanism of Token authentication, Session failure and uniform proxy of URL encryption.
The invention provides a business intelligent platform data processing system, which comprises:
the resource aggregation module is used for monitoring data of each business intelligent platform, and when one or more business intelligent platforms are monitored to have data access, the data to be accessed are pulled from the one or more business intelligent platforms by using a Push mode or a Pull mode and are aggregated, wherein the data of each business intelligent platform are stored in different storage media;
the unified authority control module is used for managing the converged data based on the RBAC model, and comprises: managing menu function authority to control users with different roles to access menu resources corresponding to the roles; managing the report authority to control users with different data authorities to access the report reading board corresponding to the data authority; managing the data authority to control different users to access different data on the report reading board;
the data security module is used for performing login verification and access authority verification on each user through SDK integration or WEB access when each user logs in and accesses one or more data in the menu resources and the report billboard, wherein an account and a password of the user are authenticated based on token authentication;
and the resource agent access module is used for performing secondary agent on the data accessed by each business intelligent platform based on Session, wherein the agent URL accessed by the user is encrypted in the secondary agent process, and the Session accessed by the user is intercepted.
As a further improvement of the invention, the resource aggregation module supports an HTTP interface protocol and an RPC interface protocol,
the resource aggregation module uses the Pull mode to Pull the data to be accessed from the one or more business intelligent platforms and performs aggregation, and the method comprises the following steps:
installing an Agent module, and registering at least two nodes to generate an index for pulling data to be accessed of one or more business intelligent platforms and provide a standard Pull interface when the one or more business intelligent platforms are not compatible with the Pull protocol;
directly pulling data to be accessed from the databases of the one or more business intelligent platforms, or calling the databases of the one or more business intelligent platforms through an API (application programming interface) interface and pulling the data to be accessed;
the data to be accessed of the one or more business intelligent platforms are pulled in turns, and the pulled data to be accessed are subjected to ETL processing to obtain processed data;
and aggregating the processed data.
As a further improvement of the invention, when the data to be accessed is directly pulled from the database of the one or more business intelligent platforms, the database address, the account number and the password of the one or more business intelligent platforms are configured,
and when the database of the one or more business intelligent platforms is called through the API interface and the data to be accessed is pulled, the interface address, the key authentication information and the request parameter of the one or more business intelligent platforms are configured.
As a further improvement of the invention, the resource aggregation module supports an HTTP interface protocol and an RPC interface protocol,
the resource aggregation module uses the Push mode to pull the data to be accessed from the one or more business intelligent platforms and performs aggregation, and the resource aggregation module comprises:
installing an Agent module, and registering at least two nodes to generate an index for pulling data to be accessed of the one or more commercial intelligent platforms and provide a standard Push interface when the one or more commercial intelligent platforms are not compatible with a Push protocol;
the one or more business intelligent platforms are accessed to the resource aggregation module, and send data to be accessed to the Agent module through a pre-configured SDK;
ETL processing is carried out on the received data to be accessed to obtain processed data;
and aggregating the processed data.
As a further improvement of the present invention, a user accesses resources of each of the business intelligent platforms through a proxy URL, and performing secondary proxy on data accessed by each of the business intelligent platforms based on Session includes:
s1, converting the proxy URL accessed by the current user into link information, wherein account numbers and passwords are added in the conversion process for interception, and HTTP 403 skipping is carried out on the link information without permission;
s2, when the current user accesses the data accessed by each business intelligent platform for the first time, generating a Session with a time window, wherein the Session with the time window comprises a user account, creation time and an IP address;
s3, creating a temporary resource URL and binding the temporary resource URL to the Session of the time window;
s4, adding a Session interceptor to prevent unauthorized users from accessing the data accessed by each business intelligent platform when the temporary resource URL or the proxy URL is leaked, wherein, when the Session with the time window of the current user expires, repeating the steps S2-S4.
The embodiment of the invention also provides a business intelligent platform data processing method, which comprises the following steps:
monitoring data of each business intelligent platform, and when one or more business intelligent platforms are monitored to have data access, pulling data to be accessed from the one or more business intelligent platforms by using a Push mode or a Pull mode and converging the data, wherein the data of each business intelligent platform are stored in different storage media;
after the data to be accessed of the one or more business intelligent platforms are converged, managing the converged data based on an RBAC model, wherein the management comprises the following steps: managing menu function authority to control users with different roles to access menu resources corresponding to the roles, managing report authority to control users with different data authorities to access report boards corresponding to the data authorities, and managing data authority to control different users to access different data on the report boards; when each user logs in to access one or more data in the menu resources and the report billboard, log-in verification and access authority verification are carried out on each user through SDK integration or WEB access, wherein an account number and a password of each user are authenticated based on token authentication;
and performing secondary proxy on the data accessed by each business intelligent platform based on Session, wherein in the secondary proxy process, proxy URL accessed by the user is encrypted, and the Session accessed by the user is intercepted.
As a further improvement of the invention, when the data are gathered, the HTTP interface protocol and the RPC interface protocol are supported, the Pull mode is used for pulling the data to be accessed from the one or more business intelligent platforms and gathering the data, and the method comprises the following steps:
installing an Agent module, and registering at least two nodes to generate an index for pulling data to be accessed of one or more business intelligent platforms and provide a standard Pull interface when the one or more business intelligent platforms are not compatible with the Pull protocol;
directly pulling data to be accessed from the databases of the one or more business intelligent platforms, or calling the databases of the one or more business intelligent platforms through an API (application programming interface) interface and pulling the data to be accessed;
the data to be accessed of the one or more business intelligent platforms are pulled in turns, and the pulled data to be accessed are subjected to ETL processing to obtain processed data;
and aggregating the processed data.
As a further improvement of the invention, when the data to be accessed is directly pulled from the database of the one or more business intelligent platforms, the database address, the account number and the password of the one or more business intelligent platforms are configured,
and when the database of the one or more business intelligent platforms is called through the API interface and the data to be accessed is pulled, the interface address, the key authentication information and the request parameter of the one or more business intelligent platforms are configured.
As a further improvement of the invention, when the data are gathered, the HTTP interface protocol and the RPC interface protocol are supported, the Push mode is used for pulling the data to be accessed from the one or more business intelligent platforms and gathering the data, and the method comprises the following steps:
installing an Agent module, and registering at least two nodes to generate an index for pulling data to be accessed of the one or more commercial intelligent platforms and provide a standard Push interface when the one or more commercial intelligent platforms are not compatible with a Push protocol;
the one or more business intelligent platforms are accessed to the resource aggregation module and send data to be accessed to the Agent module through a pre-configured SDK;
ETL processing is carried out on the received data to be accessed to obtain processed data;
and aggregating the processed data.
As a further improvement of the present invention, a user accesses resources of each of the business intelligent platforms through a proxy URL, and performing secondary proxy on data accessed by each of the business intelligent platforms based on Session includes:
s1, converting the proxy URL accessed by the current user into link information, wherein account numbers and passwords are added in the conversion process for interception, and HTTP 403 skipping is carried out on the link information without permission;
s2, when the current user accesses the data accessed by each business intelligent platform for the first time, generating a Session with a time window, wherein the Session with the time window comprises a user account, creation time and an IP address;
s3, creating a temporary resource URL and binding the temporary resource URL to the Session of the time window;
s4, adding a Session interceptor to prevent unauthorized users from accessing the data accessed by each business intelligent platform when the temporary resource URL or the proxy URL is leaked, wherein, when the Session with the time window of the current user expires, repeating the steps S2-S4.
The invention has the beneficial effects that: a plurality of BI platforms can share the unified authority control module, so that the problems of disordered authorization and high data management cost are solved; by the design of plug-in data resource aggregation standardization and the simultaneous support of two resource synchronization modes of a Push mode and a Pull mode, the problem of adaptation of authority management of a plurality of BI platforms is solved; the data security problem is solved based on a triple protection mechanism of Token authentication, Session invalidation and URL encryption unified proxy.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below. It is obvious that the drawings in the following description are only some embodiments of the invention, and that for a person skilled in the art, other drawings can be derived from them without inventive effort.
FIG. 1 is a block diagram of a business intelligence platform data processing system in accordance with an exemplary embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be obtained by a person skilled in the art without inventive step based on the embodiments of the present invention, are within the scope of protection of the present invention.
It should be noted that, if directional indications (such as up, down, left, right, front, and back … …) are involved in the embodiment of the present invention, the directional indications are only used to explain the relative positional relationship between the components, the movement situation, and the like in a specific posture (as shown in the drawing), and if the specific posture is changed, the directional indications are changed accordingly.
In addition, in the description of the present invention, the terms used are for illustrative purposes only and are not intended to limit the scope of the present invention. The terms "comprises" and/or "comprising" are used to specify the presence of stated elements, steps, operations, and/or components, but do not preclude the presence or addition of one or more other elements, steps, operations, and/or components. The terms "first," "second," and the like may be used to describe various elements, not necessarily order, and not necessarily limit the elements. In addition, in the description of the present invention, "a plurality" means two or more unless otherwise specified. These terms are only used to distinguish one element from another. These and/or other aspects will become apparent to those of ordinary skill in the art in view of the following drawings, and the description of the embodiments of the present invention will be more readily understood by those of ordinary skill in the art. The drawings are only for purposes of illustrating the described embodiments of the invention. One skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods illustrated in the present application may be employed without departing from the principles described in the present application.
As shown in fig. 1, a business intelligence platform data processing system according to an embodiment of the present invention includes:
the resource aggregation module is used for monitoring data of each business intelligent platform, and when one or more business intelligent platforms are monitored to have data access, the data to be accessed are pulled from the one or more business intelligent platforms by using a Push mode or a Pull mode and are aggregated, wherein the data of each business intelligent platform are stored in different storage media;
the unified authority control module is used for managing the converged data based on the RBAC model, and comprises: managing menu function authority to control users with different roles to access menu resources corresponding to the roles; managing the report authority to control users with different data authorities to access the report reading board corresponding to the data authority; managing the data authority to control different users to access different data on the report reading board;
the data security module is used for performing login verification and access authority verification on each user through SDK integration or WEB access when each user logs in and accesses one or more data in the menu resources and the report billboard, wherein an account and a password of the user are authenticated based on token authentication;
and the resource agent access module is used for performing secondary agent on the data accessed by each business intelligent platform based on Session, wherein the agent URL accessed by the user is encrypted in the secondary agent process, and the Session accessed by the user is intercepted.
The Business Intelligence platform in the system of the present invention is hereinafter referred to as BI platform (Business Intelligence, BI for short), and FIG. 1 is illustrated as a third party BI platform. The BI platforms in the invention can be Cognos, OBI, BO and the like, can be various business analysis tools such as Tableau, Qlikview, Power BI, FineBI and the like, can be Jaspersoft, SpagoBI, Pentaho and the like, and various BI platforms can meet BI requirements in different scenes.
The system of the invention carries out unified management on the authorities of a plurality of BI platforms by arranging a unified authority control module, thereby avoiding the problems of authorization confusion and high data management cost caused by that a plurality of BI platforms deploy authorities for management independently. The multiple BI platforms can be deployed at high availability and multiple points, data of each BI platform is stored in different storage media, the resource aggregation module is a plug-in type BI resource synchronous aggregation module, data from different BI platforms can be pulled from different storage media and aggregated, and the module can be adapted to the authorities of the multiple BI platforms, so that the data resources of each BI platform can be aggregated to the module in a unified manner, and unified management of multi-source data is realized. The system of the invention is based on Token authentication when logging in and verifying the user, encrypts the proxy URL accessed by the user in the secondary proxy process, and intercepts the Session accessed by the user, thereby forming a triple protection mechanism of Token authentication, Session invalidation and URL encryption unified proxy and ensuring the safety in the data access process.
The resource aggregation module is mainly used for realizing monitoring access of third-party BI platform data, protocol management among a plurality of BI platforms, data synchronous monitoring of the plurality of BI platforms and the like, simultaneously provides two interface protocols of HTTP and RPC, supports a Push mode and a Pull mode, and can use the Push mode or the Pull mode to perform unified aggregation on the data of the plurality of BI platforms. It is understood that when pulling data for multiple BI platforms, the manner of pulling data for each BI platform may be the same or different, e.g., using Push mode to Pull data from a first BI platform and Pull mode to Pull data from a second BI platform; pulling data from the first BI platform and the second BI platform, for example, using a Push mode; for example, data is pulled from a first BI platform and a second BI platform using Pull mode. The module supports plug-in extension of authority resource information for different BI platforms, wherein a Push mode or a Pull mode supports three modes of direct database acquisition, interface reporting and interface calling, and data aggregation of most BI platforms is covered.
The unified authority control module is mainly used for newly adding and editing user accounts, managing role department information, applying for resources (acquiring data) and the like. The authority design model of the unified authority Control module is an RBAC model (namely, a Role-Based Access Control module), the core of the model is that a Role concept is introduced between a user and an authority, the direct association of the user and the authority is cancelled, and the user authority is indirectly endowed by a method of associating the Role with the authority by the user instead, so that the aim of decoupling the user and the authority is fulfilled. The unified authority control module needs to realize authority management among users, roles, resources and the like, and aiming at the system, the module mainly needs to manage the authority of three parts of resources:
(a) the menu function authority mainly controls users with different roles to access different menu resources;
(b) the report authority mainly controls users with different data authorities to access different report boards;
(c) the data authority mainly controls different users to see different data on the report reading board, thereby achieving the isolation of bottom data.
The data security module is mainly used for managing the account and the password of a user, logging in and checking the user, checking the access authority when accessing data, checking the access authority when accessing a menu function and the like. A user can access one or more data such as menu resources, report boards and the like through SDK integration or WEB, wherein the SDK integration comprises the functions of token authentication, authority authentication, resource acquisition, access log record and the like, and the data integration is provided for a third-party BI platform.
And the resource agent access module is used for uniformly performing secondary agent on the data of the BI platforms from different sources so as not to reveal original resource links. The proxy link is ensured by a Session mechanism, proxy URL accessed by the user is encrypted in the process of secondary proxy, and the Session accessed by the user is intercepted, so that a double protection mechanism in the process of secondary proxy is formed.
In an optional embodiment, the system further includes a data management module, configured to enable a data creator to publish data, so that a data user can preview the published data, and initiate an authority application, so that after the authority application is passed, one or more data in the menu resource and the report billboard can be accessed according to corresponding authority.
An optional embodiment, where the resource aggregation module supports an HTTP interface protocol and an RPC interface protocol, and the resource aggregation module uses the Pull mode to Pull and aggregate data to be accessed from the one or more business intelligence platforms, the method includes:
installing an Agent module, and registering at least two nodes to generate an index for pulling data to be accessed of one or more business intelligent platforms and provide a standard Pull interface when the one or more business intelligent platforms are not compatible with the Pull protocol;
directly pulling data to be accessed from the databases of the one or more business intelligent platforms, or calling the databases of the one or more business intelligent platforms through an API (application programming interface) interface and pulling the data to be accessed;
the data to be accessed of the one or more business intelligent platforms are pulled in turns, and the pulled data to be accessed are subjected to ETL processing to obtain processed data;
and aggregating the processed data.
In an optional embodiment, when the data to be accessed is directly pulled from the database of the one or more business intelligent platforms, the database address, the account number and the password of the one or more business intelligent platforms are configured,
and when the database of the one or more business intelligent platforms is called through the API interface and the data to be accessed is pulled, the interface address, the key authentication information and the request parameter of the one or more business intelligent platforms are configured.
The Pull mode of the system mainly comprises the following steps:
the first step is as follows: installing an Agent module, registering at least two nodes, and because the data of each business intelligent platform is stored in different storage media and each business intelligent platform has the condition of incompatibility with the Pull protocol, developing the corresponding Agent of an Exporter, supporting the pulling of indexes of the systems and providing a standard Pull interface; when the RPC interface protocol is used, the registration center can use Nacos, Zookeeper, Eureka, a database and the like; when an HTTP interface protocol is used, the HTTP interface protocol needs proxy service and is deployed more actively;
the second step is that: data pulling is carried out on the basis of an interface protocol:
when the data access method is a database pulling method, configuring data resource related information (database address, account number and password) of a BI platform, and pulling data resources (namely data to be accessed) in SQL (database) by using the data resource related information;
when the API pull mode is adopted, interface address information, secret key authentication information and request parameter information of a third party BI platform are configured;
the third step: performing round training to pull the data resources of the BI platform, performing data warehouse technical processing (ETL for short) on the obtained data resources, and unifying the data resources into a standard data resource protocol;
the fourth step: and the data resources are sent to a Server end (namely a resource aggregation module) for aggregation, and the failure sending supports the retry and the alarm.
An optional embodiment, where the resource aggregation module supports an HTTP interface protocol and an RPC interface protocol, and the resource aggregation module uses the Push mode to pull and aggregate data to be accessed from the one or more business intelligence platforms, the method includes:
installing an Agent module, and registering at least two nodes to generate an index for pulling data to be accessed of the one or more commercial intelligent platforms and provide a standard Push interface when the one or more commercial intelligent platforms are not compatible with a Push protocol;
the one or more business intelligent platforms are accessed to the resource aggregation module, and send data to be accessed to the Agent module through a pre-configured SDK;
ETL processing is carried out on the received data to be accessed to obtain processed data;
and aggregating the processed data.
The Push mode of the system mainly comprises the following steps:
the first step is as follows: installing an Agent module, registering at least two nodes, and because the data of each business intelligent platform is stored in different storage media and each business intelligent platform has the condition of incompatibility with the Pull protocol, developing the corresponding Agent of an Exporter, supporting the pulling of indexes of the systems and providing a standard Pull interface; when the RPC interface protocol is used, the registration center can use Nacos, Zookeeper, Eureka, a database and the like; when an HTTP interface protocol is used, the HTTP interface protocol needs proxy service and multi-active deployment;
the second step is that: providing SDK Client and data transmission key information;
the third step: accessing a third party BI platform, and sending data to an Agent module through an SDK Client;
the fourth step: ETL processing is carried out on the obtained data resources (namely the data to be accessed), and a standard data resource protocol is unified;
the fifth step: and sending the resource data to a Server (namely a resource aggregation module) for aggregation, and sending failure support retry and alarm.
In an alternative embodiment, a user accesses a resource of each of the business intelligence platforms through a proxy URL, and performing secondary proxy on data accessed by each of the business intelligence platforms based on Session includes:
s1, converting the proxy URL accessed by the current user into link information, wherein account numbers and passwords are added in the conversion process for interception, and HTTP 403 skipping is carried out on the link information without permission;
s2, when the current user accesses the data accessed by each business intelligent platform for the first time, generating a Session with a time window, wherein the Session with the time window comprises a user account, a creation time and an IP address;
s3, creating a temporary resource URL and binding the temporary resource URL to the Session of the time window;
s4, adding a Session interceptor to prevent unauthorized users from accessing the data accessed by each business intelligent platform when the temporary resource URL or the proxy URL is leaked, wherein, when the Session with the time window of the current user expires, repeating the steps S2-S4.
The system of the invention uniformly carries out secondary proxy aiming at data resources from different sources (namely, data accessed by different BI platforms), and aims to avoid revealing original resource links. The proxy link is guaranteed by a Session mechanism, for example, when a user (Zusanli) opens the data resource of the BI platform A and accesses through the proxy URL A, when the proxy URL A is leaked, because the proxy URL A binds the Session of Zusanli (Session with a time window), other users cannot use the Session, and the data security of the BI platform A is guaranteed. The main implementation steps of the secondary proxy are as follows:
the first step is as follows: data resource URL proxy processing: and converting the proxy URL accessed by the current user into link information which cannot be identified, adding an account and a password interception mechanism to the accessed data resource, and performing HTTP 403 skipping if the access is not authorized.
The second step is that: and generating a Session with a time window by the first access of the user, wherein the Session comprises a user account, creation time, an IP address and the like.
The third step: a temporary resource URL is created and bound into the Session with the time window that has been generated.
The fourth step: and a Session interception mechanism is added, and if the temporary resource URL or the proxy URL is leaked, other users without permission cannot access the data resource. And if the Session of the browser of the current user expires, the second step, the third step and the fourth step need to be carried out again.
The invention discloses a business intelligent platform data processing method, which comprises the following steps:
monitoring data of each business intelligent platform, and when one or more business intelligent platforms are monitored to have data access, pulling data to be accessed from the one or more business intelligent platforms by using a Push mode or a Pull mode and converging the data, wherein the data of each business intelligent platform are stored in different storage media;
after the data to be accessed of the one or more business intelligent platforms are converged, managing the converged data based on an RBAC model, wherein the management comprises the following steps: managing menu function authority to control users with different roles to access menu resources corresponding to the roles; managing the report authority to control users with different data authorities to access the report reading board corresponding to the data authority; managing the data authority to control different users to access different data on the report reading board; when each user logs in to access one or more data in the menu resources and the report billboard, log-in verification and access authority verification are carried out on each user through SDK integration or WEB access, wherein an account number and a password of each user are authenticated based on token authentication;
and performing secondary proxy on the data accessed by each business intelligent platform based on Session, wherein in the secondary proxy process, proxy URL accessed by the user is encrypted, and the Session accessed by the user is intercepted.
An optional embodiment, which supports an HTTP interface protocol and an RPC interface protocol when aggregating data, and uses the Pull mode to Pull data to be accessed from the one or more business intelligent platforms and perform aggregation, includes:
installing an Agent module, and registering at least two nodes to generate an index for pulling data to be accessed of one or more business intelligent platforms and provide a standard Pull interface when the one or more business intelligent platforms are not compatible with the Pull protocol;
directly pulling data to be accessed from the databases of the one or more business intelligent platforms, or calling the databases of the one or more business intelligent platforms through an API (application programming interface) interface and pulling the data to be accessed;
the data to be accessed of the one or more business intelligent platforms are pulled in turns, and the pulled data to be accessed are subjected to ETL processing to obtain processed data;
and aggregating the processed data.
In an optional embodiment, when the data to be accessed is directly pulled from the database of the one or more business intelligent platforms, the database address, the account number and the password of the one or more business intelligent platforms are configured,
and when the database of the one or more business intelligent platforms is called through the API interface and the data to be accessed is pulled, the interface address, the key authentication information and the request parameter of the one or more business intelligent platforms are configured.
An optional embodiment, which supports an HTTP interface protocol and an RPC interface protocol when aggregating data, and uses the Push mode to pull data to be accessed from the one or more business intelligent platforms and perform aggregation, includes:
installing an Agent module, and registering at least two nodes to generate an index for pulling data to be accessed of one or more commercial intelligent platforms and provide a standard Push interface when the one or more commercial intelligent platforms are not compatible with a Push protocol;
the one or more business intelligent platforms are accessed to the resource aggregation module and send data to be accessed to the Agent module through a pre-configured SDK;
ETL processing is carried out on the received data to be accessed to obtain processed data;
and aggregating the processed data.
In an alternative embodiment, a user accesses a resource of each of the business intelligence platforms through a proxy URL, and performing secondary proxy on data accessed by each of the business intelligence platforms based on Session includes:
s1, converting the proxy URL accessed by the current user into link information, wherein account numbers and passwords are added in the conversion process for interception, and HTTP 403 skipping is carried out on the link information without permission;
s2, when the current user accesses the data accessed by each business intelligent platform for the first time, generating a Session with a time window, wherein the Session with the time window comprises a user account, creation time and an IP address;
s3, creating a temporary resource URL and binding the temporary resource URL to the Session of the time window;
s4, adding a Session interceptor to prevent unauthorized users from accessing the data accessed by each business intelligent platform when the temporary resource URL or the proxy URL is leaked, wherein, when the Session with the time window of the current user expires, repeating the steps S2-S4.
In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the invention may be practiced without these specific details. In some instances, it is not
Well-known methods, structures and techniques have been shown in detail in order not to obscure an understanding of this description.
Furthermore, those of ordinary skill in the art will appreciate that while some embodiments described herein include some features included in other embodiments, rather than other features, combinations of features of different embodiments are meant to be within the scope of the invention and form different embodiments. For example, in the claims, any of the claimed embodiments may be used in any combination.
It will be understood by those skilled in the art that while the present invention has been described with reference to exemplary embodiments, various changes may be made and equivalents may be substituted for elements thereof without departing from the scope of the invention. In addition, many modifications may be made to adapt a particular situation or material to the teachings of the invention without departing from the essential scope thereof. Therefore, it is intended that the invention not be limited to the particular embodiment disclosed, but that the invention will include all embodiments falling within the scope of the appended claims.

Claims (8)

1. A business intelligence platform data processing system, the system comprising:
the resource aggregation module is used for monitoring data of each business intelligent platform, and when one or more business intelligent platforms are monitored to have data access, the data to be accessed are pulled from the one or more business intelligent platforms by using a Push mode or a Pull mode and are aggregated, wherein the data of each business intelligent platform are stored in different storage media;
the unified authority control module is used for managing the converged data based on an RBAC model, and is used for: managing menu function authority to control users with different roles to access menu resources corresponding to the roles; managing the report authority to control users with different data authorities to access the report reading board corresponding to the data authority; managing the data authority to control different users to access different data on the report reading board;
the data security module is used for performing login verification and access authority verification on each user through SDK integration or WEB access when each user logs in and accesses one or more data in the menu resources and the report billboard, wherein an account and a password of the user are authenticated based on token authentication;
the resource agent access module is used for carrying out secondary agent on the data accessed by each business intelligent platform based on Session, wherein the agent URL accessed by the user is encrypted in the secondary agent process, and the Session accessed by the user is intercepted;
the user accesses the resources of each of the business intelligence platforms via the proxy URL,
the Session-based secondary proxy of the data accessed by each business intelligent platform comprises:
s1, converting the proxy URL accessed by the current user into link information, wherein account numbers and passwords are added in the conversion process for interception, and HTTP 403 skipping is carried out on the link information without permission;
s2, when the current user accesses the data accessed by each business intelligent platform for the first time, generating a Session with a time window, wherein the Session with the time window comprises a user account, creation time and an IP address;
s3, creating a temporary resource URL and binding the temporary resource URL to the Session of the time window;
s4, adding a Session interceptor to prevent unauthorized users from accessing the data accessed by each business intelligent platform when the temporary resource URL or the proxy URL is leaked, wherein, when the Session with the time window of the current user expires, repeating the steps S2-S4.
2. The system of claim 1, wherein the resource aggregation module supports an HTTP interface protocol and an RPC interface protocol,
the resource aggregation module uses the Pull mode to Pull the data to be accessed from the one or more business intelligent platforms and performs aggregation, and the method comprises the following steps:
installing an Agent module, and registering at least two nodes to generate an index for pulling data to be accessed of one or more business intelligent platforms and provide a standard Pull interface when the one or more business intelligent platforms are not compatible with the Pull protocol;
directly pulling data to be accessed from the databases of the one or more business intelligent platforms, or calling the databases of the one or more business intelligent platforms through an API (application programming interface) interface and pulling the data to be accessed;
polling and pulling the data to be accessed of the one or more business intelligent platforms, and carrying out ETL (extract transform load) processing on the pulled data to be accessed to obtain processed data;
and aggregating the processed data.
3. The system of claim 2, wherein the database address, account number, and password of the one or more business intelligence platforms are configured when the data to be accessed is pulled directly from the database of the one or more business intelligence platforms,
and when the database of the one or more business intelligent platforms is called through the API interface and the data to be accessed is pulled, the interface address, the key authentication information and the request parameter of the one or more business intelligent platforms are configured.
4. The system of claim 1, wherein the resource aggregation module supports an HTTP interface protocol and an RPC interface protocol,
the resource aggregation module uses the Push mode to pull the data to be accessed from the one or more business intelligent platforms and performs aggregation, and the resource aggregation module comprises:
installing an Agent module, and registering at least two nodes to generate an index for pulling data to be accessed of the one or more commercial intelligent platforms and provide a standard Push interface when the one or more commercial intelligent platforms are not compatible with a Push protocol;
the one or more business intelligent platforms are accessed to the resource aggregation module, and send data to be accessed to the Agent module through a pre-configured SDK;
ETL processing is carried out on the received data to be accessed to obtain processed data;
and aggregating the processed data.
5. A business intelligence platform data processing method, the method comprising:
monitoring data of each business intelligent platform, and when one or more business intelligent platforms are monitored to have data access, pulling data to be accessed from the one or more business intelligent platforms by using a Push mode or a Pull mode and converging the data, wherein the data of each business intelligent platform are stored in different storage media;
after the data to be accessed of the one or more business intelligent platforms are aggregated, managing the aggregated data based on an RBAC model, wherein the management comprises the following steps: managing menu function authority to control users with different roles to access menu resources corresponding to the roles; managing the report authority to control users with different data authorities to access the report reading board corresponding to the data authority; managing the data authority to control different users to access different data on the report reading board; when each user logs in to access one or more data in the menu resources and the report billboard, log-in verification and access authority verification are carried out on each user through SDK integration or WEB access, wherein an account number and a password of each user are authenticated based on token authentication;
performing secondary proxy on data accessed by each business intelligent platform based on Session, wherein proxy URLs accessed by users are encrypted in the secondary proxy process, and the Session accessed by the users is intercepted;
the user accesses the resource of each business intelligent platform through the proxy URL, and the secondary proxy of the data accessed by each business intelligent platform based on the Session comprises the following steps:
s1, converting the proxy URL accessed by the current user into link information, wherein account numbers and passwords are added in the conversion process for interception, and HTTP 403 skipping is carried out on the link information without permission;
s2, when the current user accesses the data accessed by each business intelligent platform for the first time, generating a Session with a time window, wherein the Session with the time window comprises a user account, creation time and an IP address;
s3, creating a temporary resource URL and binding the temporary resource URL to the Session of the time window;
s4, adding a Session interceptor to prevent unauthorized users from accessing the data accessed by each business intelligent platform when the temporary resource URL or the proxy URL is leaked, wherein, when the Session with the time window of the current user expires, repeating the steps S2-S4.
6. The method of claim 5, wherein aggregating data supports an HTTP interface protocol and an RPC interface protocol, and wherein using the Pull mode to Pull data to be accessed from the one or more business intelligence platforms for aggregation comprises:
installing an Agent module, and registering at least two nodes to generate an index for pulling data to be accessed of one or more business intelligent platforms and provide a standard Pull interface when the one or more business intelligent platforms are not compatible with the Pull protocol;
directly pulling data to be accessed from the databases of the one or more business intelligent platforms, or calling the databases of the one or more business intelligent platforms through an API (application programming interface) interface and pulling the data to be accessed;
polling and pulling the data to be accessed of the one or more business intelligent platforms, and carrying out ETL (extract transform load) processing on the pulled data to be accessed to obtain processed data;
and aggregating the processed data.
7. The method of claim 6, wherein the database address, account number, and password of the one or more business intelligence platforms are configured when the data to be accessed is pulled directly from the database of the one or more business intelligence platforms,
and when the database of the one or more business intelligent platforms is called through the API interface and the data to be accessed is pulled, the interface address, the key authentication information and the request parameter of the one or more business intelligent platforms are configured.
8. The method of claim 5, wherein the supporting HTTP interface protocol and RPC interface protocol when aggregating data, using the Push mode to pull and aggregate data to be accessed from the one or more business intelligence platforms, comprises:
installing an Agent module, and registering at least two nodes to generate an index for pulling data to be accessed of the one or more commercial intelligent platforms and provide a standard Push interface when the one or more commercial intelligent platforms are not compatible with a Push protocol;
the one or more business intelligent platforms are accessed to the resource aggregation module and send data to be accessed to the Agent module through a pre-configured SDK;
ETL processing is carried out on the received data to be accessed to obtain processed data;
and aggregating the processed data.
CN202111248721.0A 2021-10-26 2021-10-26 Business intelligent platform data processing system and method Active CN113973017B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202111248721.0A CN113973017B (en) 2021-10-26 2021-10-26 Business intelligent platform data processing system and method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202111248721.0A CN113973017B (en) 2021-10-26 2021-10-26 Business intelligent platform data processing system and method

Publications (2)

Publication Number Publication Date
CN113973017A CN113973017A (en) 2022-01-25
CN113973017B true CN113973017B (en) 2022-06-07

Family

ID=79588729

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202111248721.0A Active CN113973017B (en) 2021-10-26 2021-10-26 Business intelligent platform data processing system and method

Country Status (1)

Country Link
CN (1) CN113973017B (en)

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104580211B (en) * 2015-01-08 2018-02-23 浪潮软件集团有限公司 SOA architecture-based intrusive system
CN106612246A (en) * 2015-10-21 2017-05-03 星际空间（天津）科技发展有限公司 Unified authentication method for simulation identity
CN108476216B (en) * 2016-03-31 2021-01-22 甲骨文国际公司 System and method for integrating a transactional middleware platform with a centralized access manager for single sign-on in an enterprise-class computing environment
CN112905918A (en) * 2021-03-06 2021-06-04 上海数依数据科技有限公司 Data service convergence engine and management method thereof

Also Published As

Publication number Publication date
CN113973017A (en) 2022-01-25

Similar Documents

Publication Publication Date Title
US20220124081A1 (en) System for Managing Remote Software Applications
US9130920B2 (en) Monitoring of authorization-exceeding activity in distributed networks
US10484385B2 (en) Accessing an application through application clients and web browsers
CN102638454B (en) Plug-in type SSO (single signon) integration method oriented to HTTP (hypertext transfer protocol) identity authentication protocol
US8990911B2 (en) System and method for single sign-on to resources across a network
EP2257026B1 (en) System and method for accessing private digital content
US10084790B2 (en) Peer to peer enterprise file sharing
CN107122674B (en) Access method of oracle database applied to operation and maintenance auditing system
US20130117554A1 (en) User key management for the Secure Shell (SSH)
CN112468481B (en) Single-page and multi-page web application identity integrated authentication method based on CAS
CN109379363B (en) A kind of single-sign-on integrated approach and system based on intensive platform
CN111314340A (en) Authentication method and authentication platform
US11153293B1 (en) Identity information linking
CN109150800A (en) Login access method, system and storage medium
CN109462595A (en) Data-interface secure exchange method based on RestFul
CN103905395A (en) WEB access control method and system based on redirection
CN116170234A (en) Single sign-on method and system based on virtual account authentication
CN105100068A (en) System and method for realizing single sign-on
CN113973017B (en) Business intelligent platform data processing system and method
US10735399B2 (en) System, service providing apparatus, control method for system, and storage medium
CN114070616A (en) Distributed session sharing method and system based on redis cache
CN110741371B (en) Information processing apparatus, protection processing apparatus, and use terminal
KR20190019317A (en) Server and method for authentication in on-demand SaaS aggregation service platform
CN106790026B (en) Hadoop-based multi-tenant network disk authentication method and system
CN117527414A (en) System single sign-on method, device, equipment and medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant