CN115242879A

CN115242879A - Data sharing system and method

Info

Publication number: CN115242879A
Application number: CN202210764896.5A
Authority: CN
Inventors: 赵晓芬; 李林; 张博
Original assignee: Inspur Communication Technology Co Ltd
Current assignee: Inspur Communication Technology Co Ltd
Priority date: 2022-06-29
Filing date: 2022-06-29
Publication date: 2022-10-25
Anticipated expiration: 2042-06-29
Also published as: CN115242879B

Abstract

The invention provides a data sharing system and a method, wherein the system comprises: the system comprises a sharing interface module, a data warehouse management module, a data processing module and a visualization management module; the sharing interface module is used for determining a corresponding sharing service interface according to the requirement of the upper application; the data warehouse management module is used for sending an access request of a service requester based on an interaction mode of various custom standardized interfaces and a shared service interface corresponding to an upper application; the data processing module is used for selecting a target server cluster based on a load balancing algorithm and cluster performance weights of different server clusters and returning response data; and providing the visual interface display response data to the service requester. The invention manages the access mode of the system in a unified way through different standardized interfaces, adopts the server cluster, selects the server cluster with matched performance for the access requests of different service requesters with different service types, and improves the expansion performance of the data sharing service.

Description

Data sharing system and method

Technical Field

The present invention relates to the field of data sharing technologies, and in particular, to a data sharing system and method.

Background

With the rapid development of the internet and the information technology, particularly under the background of big data, all industries have own information application platforms or big data platforms, and the data summarization and sharing requirements are very large.

In view of the requirement diversity of the client of the system application for data sharing and the frequency of the requirement change, the sharing mode and the sharing interface need to be frequently modified, which has certain difficulty for the system platform developer, especially for the operation and maintenance personnel.

How to effectively utilize the data sharing platform and help operation and maintenance personnel to efficiently solve the requirement of a client on the aspect of high-concurrency data sharing of the platform is of great significance.

Disclosure of Invention

The invention provides a data sharing system and a data sharing method, which are used for solving the problems that in the prior art, a client needs to frequently modify a sharing mode and a sharing interface for the requirement diversity and the requirement change of data sharing, the operation and the management are inconvenient, and proper data sharing service cannot be provided according to corresponding requirements in real time.

The existing information application platform or data sharing platform has diversity to the data sharing requirements of different application clients and the frequency of requirement change, cannot provide flexible support, needs frequent modification of sharing modes and sharing interfaces, has certain operation difficulty in actual operation and maintenance, and is easy to make mistakes.

The invention provides a data sharing system, comprising: the system comprises a sharing interface module, a data warehouse management module, a data processing module and a visualization management module;

the shared interface module is used for determining a shared service interface corresponding to each upper layer application according to the requirements of the upper layer applications;

the data warehouse management module is used for sending an access request of a service requester based on interaction modes of various user-defined standardized interfaces and the shared service interface corresponding to the upper application; the interaction modes of the multiple user-defined standardized interfaces comprise a message interaction mode, a publish-subscribe mode and a database access mode;

the data processing module is used for selecting a target server cluster for processing the access request of the service requester based on a load balancing algorithm and the determined cluster performance weights of different server clusters in the server clusters, and returning corresponding response data by the target server cluster;

the visualization management module is used for providing a visualization interface for the service requester to display the response data based on the response data obtained by the data processing module.

According to the data sharing system provided by the present invention, the shared interface module is configured to determine, according to a requirement of an upper layer application, a shared service interface corresponding to each upper layer application, and includes at least one of:

controlling whether the third-party application is accessed to the data sharing system or not based on the access authority corresponding to the third-party application, and providing a channel encryption mechanism;

or determining an available interface of each tenant in the multiple tenants based on the access right and the access quantity requirement of the multiple tenants;

or determining whether to limit the access of the third-party application based on the access frequency of the third-party application;

or, based on the priority order of the access requests in the access queue and the connection state of the access requests, adjusting the processing priority of the access requests with different priorities.

According to a data sharing system provided by the present invention, the sending the access request to the data processing module based on any one or a combination of a plurality of manners of a message interaction manner, a publish-subscribe manner, and a database manner includes:

if the interactive modes of the multiple user-defined standardized interfaces are determined to be message interactive modes, determining that the corresponding response mode in the message interactive modes is synchronous response or asynchronous response based on the data volume of the access request of the service requester, the response time requirement and the corresponding response performance index;

if the interactive mode of the multiple user-defined standardized interfaces is determined to be a publish-subscribe mode, acquiring response data corresponding to a registration request based on the registration request of a service requester;

and if the interactive mode of the plurality of user-defined standardized interfaces is determined to be a database mode, determining an access interface of an external system to the database based on an ODBC or JDBC access mode.

According to a data sharing system provided by the present invention, if it is determined that the interaction modes of the plurality of customized standardized interfaces are message interaction modes, after determining that the corresponding response mode in the message interaction modes is a synchronous response or an asynchronous response based on the data size of the access request of the service requester, the response time requirement, and the corresponding response performance index, the method includes:

if the corresponding response mode in the message interaction mode is synchronous response, the service requester enters a blocking state to wait for the response of the service provider;

and if the corresponding response mode in the message interaction mode is asynchronous response, the service requester determines whether the response data is prepared or not through polling or event notification.

According to a data sharing system provided by the present invention, if it is determined that the interaction manner of the plurality of customized standardized interfaces is a publish-subscribe manner, acquiring response data corresponding to a registration request based on the registration request of a service requester, includes:

the service provider determining a readiness state of the response data;

and if the service provider determines that the preparation state of the response data is successful, periodically feeding back the response data to the service requester according to a feedback data mode and a feedback period included in the registration request.

According to a data sharing system provided by the present invention, if the access request is used for data query, the sending of the access request to the data processing module based on any one of KPI data query, data model structure query, and DQI data query includes:

determining a data query result corresponding to KPI data query based on dimension attributes and query indexes in KPI data query and a Web Service/Simple Object Access Protocol (SOAP) technology;

or, determining a data query result corresponding to the data model structure query based on the data structure name, the dimension index and the Web Service/SOAP technology in the data model structure query;

or, determining a data query result corresponding to the DQI data query based on the dimension attribute and the query index in the DQI data query and the Web Service/SOAP technology.

According to a data sharing system provided by the present invention, the data processing module is configured to select a target server cluster for processing the access request of the service requester based on a load balancing algorithm and a determined cluster performance weight of different server clusters in the server clusters, and the target server cluster returns corresponding response data, and the data processing system includes:

determining one or more matched pending server clusters in all the service clusters based on the service type, data volume or response performance requirement corresponding to the access request of the service requester;

based on a load balancing algorithm and the cluster performance weight of each undetermined server cluster, selecting a target service cluster to process the access request of the service requester;

and sending the corresponding response data returned by the target server cluster to the service requester.

According to the data sharing system provided by the invention, the system further comprises a configuration management module, a display management module and a display management module, wherein the configuration management module is used for carrying out customized configuration on all displayable units included in the visualization management module; wherein all displayable units include service maps, subscription management, service management, audit management, and shared log management.

The invention provides a data sharing method, which comprises the following steps:

determining a shared service interface corresponding to each upper-layer application according to the requirements of the upper-layer applications;

sending an access request of a service requester based on an interactive mode of a plurality of user-defined standardized interfaces and the shared service interface corresponding to the upper application; the interaction modes of the multiple user-defined standardized interfaces comprise a message interaction mode, a publish-subscribe mode and a database access mode;

based on a load balancing algorithm and the determined cluster performance weights of different server clusters in the server clusters, selecting a target server cluster for processing the access request of the service requester, and returning corresponding response data by the target server cluster;

and providing a visual interface for the service requester to display the response data based on the response data obtained by the data processing module.

The data sharing system and the method provided by the invention uniformly manage the mode of accessing the data sharing system by an external system through different standardized interfaces, adopt the server cluster, select the server cluster with matched performance for the access requests of different service requesters with different service types, and improve the expansion performance of the data sharing service of the data sharing system. On the premise of ensuring data safety and reliability, the data sharing is realized more flexibly.

Drawings

In order to more clearly illustrate the technical solutions of the present invention or the prior art, the drawings needed for the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and those skilled in the art can also obtain other drawings according to the drawings without creative efforts.

FIG. 1 is a schematic diagram of a data sharing system provided by the present invention;

FIG. 2 is a schematic diagram of the interaction of online request and synchronous response provided by the present invention;

FIG. 3 is one of the interaction diagrams of the online request and asynchronous response mode provided by the present invention;

FIG. 4 is a second schematic diagram illustrating interaction between online requests and asynchronous responses provided by the present invention;

FIG. 5 is an interactive diagram of a publish-subscribe approach provided by the present invention;

fig. 6 is a schematic flow chart of a data sharing method provided by the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is obvious that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

The data sharing system and method of the present invention are described below in conjunction with fig. 1-6.

FIG. 1 is a schematic structural diagram of a data sharing system provided by the present invention; as shown in fig. 1, the data sharing system 100 includes: a shared interface module 110, a data warehouse management module 120, a data processing module 130, and a visualization management module 140;

the shared interface module is used for determining a shared service interface corresponding to each upper-layer application according to the requirements of the upper-layer applications;

the data warehouse management module is used for sending an access request of a service requester based on interaction modes of various custom standardized interfaces and the shared service interface corresponding to the upper application; the interaction modes of the multiple customized standardized interfaces comprise a message interaction mode, a publish-subscribe mode and a database access mode;

and the visualization management module provides a visualization interface for the service requester to display the response data based on the response data obtained by the data processing module.

Specifically, the data sharing system provided by the invention can be oriented to different scenes and provide differentiated data sharing. Aiming at various applications provided by different manufacturers, data support can be provided for upper-layer applications mainly through three types of differential sharing modes. The three types of differentiated sharing modes mainly include: real-time KPI calculation class, real-time detail inquiry class and analysis class non-real-time statistics. The real-time KPI calculation class mainly provides data support for upper-layer application through a real-time signaling KPI, data quality early warning and on-time statistics; the real-time detail query class mainly provides data support for upper-layer application through number query, user record query and complaint fault query; the analysis type non-real-time statistics mainly comprises the steps of index acquisition and presentation, detail data acquisition and data support provided for upper-layer application by using application of computing/storage resources of a data sharing system.

And the service request or the upper application requirement of data support is provided through real-time KPI calculation, and the method mainly has the following characteristics: the requirement on the real-time performance of data is high, the shared data volume is small, the interaction process is generally unidirectional, and the data acquisition frequency is real-time.

The method provides a service request supported by data or an upper application requirement through a real-time detail query class, and mainly has the following characteristics: the requirement on the real-time performance of data is low, the shared data volume is small, the interaction process is usually bidirectional, and the data acquisition frequency is on demand.

The service request or upper application requirement supported by data provided by analyzing class non-real-time statistics mainly has the following characteristics: the requirement on data instantaneity is low, the size of the shared data can be both, the interaction process is usually bidirectional, and the data acquisition frequency is timing.

The three types of differential sharing modes are determined according to the requirements of upper-layer application through corresponding sharing service interface modules in the data sharing system, and the specific requirements of the upper-layer application can meet any one of the characteristics of temporal KPI calculation, real-time detail query, analysis type non-real-time statistics of service requests for providing data support or upper-layer application requirements; a corresponding shared service interface is determined.

Optionally, the shared interface module is configured to determine, according to a requirement of an upper layer application, a shared service interface corresponding to each upper layer application, where the shared service interface includes at least one of:

or determining an available interface of each tenant in the multi-tenants based on the access authority and the access quantity requirement of the multi-tenants;

Specifically, after determining the corresponding shared service interface according to the characteristics of the upper application requirement, it is further required to determine the authority, access frequency, access amount, and the like corresponding to the upper application requirement, determine whether to access the shared data through the determined shared service interface, and further perform unified management on the shared service interface.

For example, the access right corresponding to the third-party application is acquired, and whether the third application has the right to access the data sharing system is determined, so that the third-party application with the access right is allowed to access the data sharing system, and the third-party application without the access right to access the data sharing system is not allowed to access. And different encryption mechanisms are set according to different weights, so that the confidentiality of data is flexibly realized.

Or, there are some servers in the data sharing system rented by multiple tenants, and these tenants have different access rights to different servers, and through the dynamic setting of the access rights to each tenant, the service of a certain tenant can be started or stopped independently without mutual influence. And can dynamically allocate and recycle the required resources according to the demands of tenants (upper application), including CPU computing resources, I/O bandwidth, memory, parallelism, space size, and time window; the management capability of the I/O bandwidth, the CPU and the memory is particularly important. And dynamic resource supply can be performed according to a Service Level Agreement (SLA) of each tenant. The SLA levels here may include: different times, different applications, different jobs, different users, etc.

Or, determining whether the third-party application is a malicious attack or not according to the access frequency of the third-party application, and if the third-party application is determined to be the malicious attack, preventing the third-party application from accessing the data sharing system, so that the performance and the safety of the data warehouse and the server are ensured. For example, if the access frequency is greater than a preset threshold, it is determined that the attack is malicious, and the preset threshold may be obtained according to a statistical value in actual application, or set according to experience.

Or, through different access queues of the device, access requests with different priorities are stored, and the connection state of the access requests in each access queue is monitored, so that the data sharing system is ensured to meet the processing delay requirement of the corresponding access request under high load. And if the connection of a certain access request is overtime, releasing the corresponding server resource.

The access quantity requirement is the number of concurrent access requests in unit time, and can support at least 50 concurrent processes; when the file mode is used for interaction, files with large data volume (for example, more than 10M) can be compressed and then transmitted.

Optionally, the data warehouse management module is configured to send an access request of a service requester based on an interaction manner of multiple customized standardized interfaces and the shared service interface corresponding to the upper application, and includes:

determining that the access request is for data sharing or data querying;

if the access request is used for data sharing, the access request is sent to the data processing module based on any one or combination of a plurality of modes of a message interaction mode, a publish-subscribe mode and a database mode;

and if the access request is used for data query, the access request is sent to the data processing module based on any one of KPI data query, data model structure query and DQI data query.

Specifically, the data sharing system provided by the invention aims to ensure the requirements of data sharing service and interfaces such as universality, usability, consistency and stability. By customizing the standardized interface and the interface protocol, any external application needing to acquire data is ensured, and the data can be conveniently acquired through the data sharing system only by following the standard requirement of the data sharing interface.

Wherein, the user-defined standardized interface and the interface protocol comprise: message queues, web services, file Transfer Protocol (FTP), database access interfaces, socket, and the like. The message queue may be divided into a master message queue, a plurality of slave message queues, and a currently processing message queue as the master message queue, wherein access requests included therein have a higher priority. And the master message queue and the slave message queue can be mutually converted. That is, the master message queue at the current time may be the slave message queue at the next time.

And the corresponding interface protocol meets the following conditions:

the whole interface message is based on XML language and conforms to XML V1.0 (2 nd Edition or highher) specification and XML Schema (1 st Edition or highher) specification.

The interface implementation conforms to the specification of Simple Object Access Protocol (SOAP) V1.1 version, the high-version SOAP Protocol ensures compatibility with 1.1 version, and the interface Description uses Web Services Description Language (WSDL).

The interface implementation supports at least transmission of SOAP over hypertext Transfer Protocol (HTTP) V1.0 Protocol.

The interface implementation conforms to the specification of the WSDL language V1.1 version, and the compatibility of the WSDL language with the high version is ensured with the 1.1 version.

The interface description supports binding of WSDL on the SOAP V1.1 protocol.

Aiming at the interaction mode of the user-defined standardized interface, the invention provides three forms: message interaction mode, publish-subscribe mode and database mode.

The message interaction mode is a mode of data interaction between a service requester and a service provider. The message interaction mode is divided into an online request and synchronous response mode and an online request and asynchronous response mode according to the data interaction mode, and the online request and asynchronous response mode respectively correspond to different data volume and response performance indexes. Typically for a single acquisition of shared data.

The publish-subscribe method is mainly that the service requester can periodically obtain subscription data.

The database mode can adopt different database connection protocols as long as aiming at different types of databases, and the invention avoids directly exposing the physical model of the database and improves the coupling of the data sharing system through the universal connection protocol.

In the above description, data sharing is mainly provided for the external system through the three standardized interface interaction modes, and of course, operations such as collecting, storing, analyzing and the like may also be performed on the access request data of the service requester.

Any system internal data needing to be shared can realize data sharing on the premise of not influencing a system data architecture, and the shared data and the system internal data are designed in a loose coupling mode. The system has perfect data abstraction and encapsulation capacity, and internal data can be encapsulated into application-oriented data sharing services. The shared data is completely separated from the internal data, and the internal data model is not changed due to the change of the shared data model.

In addition, the data warehouse management module in the data sharing system can also provide data query functions for external systems. Specifically, the data warehouse management module of the invention adopts a technology based on Web Service/Simple Object Access Protocol (SOAP) to realize the data query requirement of the external system.

And determining metadata, i.e., the smallest data unit, facilitates processing in a unified manner between different platforms or applications through the unified smallest data unit. The format of the shared data is defined by the metadata. The service requester for realizing the upper application and the third-party application can access the data according to the description information of the data content and the structure under the condition that the data organization structure in the data sharing system is unclear.

And providing a query result of the data through three preset query modes. The preset three query modes are as follows: key Performance Indicator (KPI) data queries, data model structure queries, and Design Quality Indicator (DQI) data queries.

Optionally, the sending the access request to the data processing module based on any one or a combination of multiple manners of a message interaction manner, a publish-subscribe manner, and a database manner includes:

if the multiple user-defined standardized interfaces are determined to be in a message interaction mode, determining that a corresponding response mode in the message interaction mode is a synchronous response or an asynchronous response based on the data volume of the access request of the service requester, the response time requirement and a corresponding response performance index;

if the various user-defined standardized interfaces are determined to be in a publish-subscribe mode, acquiring response data corresponding to a registration request based on the registration request of a service requester;

and if the plurality of user-defined standardized interfaces are determined to be in a database mode, determining an access interface of an external system to the database based on an ODBC or JDBC access mode.

Specifically, the interaction modes of the multiple customized standardized interfaces include a message interaction mode, a publish-subscribe mode, a database mode, and the like, and the corresponding different customized standardized interfaces correspond to different processing flows.

For example, if the interaction mode of the plurality of customized standardized interfaces is a message interaction mode, it is determined that a corresponding response mode in the message interaction mode is a synchronous response or an asynchronous response based on the data volume of the access request of the service requester, the response time requirement, and the corresponding response performance index. For example, the data volume of the access request is small, the response time requirement is short, the response performance index is high, and a synchronous response mode is generally adopted; if the data volume of the access request is large and the response performance index is low, an asynchronous response mode is generally adopted.

Or determining that the interaction mode of the plurality of user-defined standardized interfaces is a publish-subscribe mode, and acquiring response data corresponding to a registration request based on the registration request of a service requester; for example, if the registration request includes the feedback data, the service provider determines whether the service provider supports the feedback data, and if so, returns response data satisfying the feedback data according to the registration request.

If the interactive mode of the multiple user-defined standardized interfaces is determined to be a Database mode, determining an access interface of an external system to the Database based on an Open Database connection (ODBC) or Java Database connection (JDBC) access mode.

After the interaction mode of the standardized interfaces of the service requester and the service provider is determined to be a database mode, a uniform access mode can be provided for an external system through a standard database connection protocol ODBC or JDBC, and attack and tampering of a third party caused by exposure of a physical model of a database are avoided. Because the change of the physical model of the database can directly influence the upper application, the adoption of the standard database connection protocol ODBC or JDBC provided by the invention can improve the coupling degree of the data sharing system.

Optionally, if it is determined that the plurality of customized standardized interfaces are in a message interaction manner, after determining that a corresponding response manner in the message interaction manner is a synchronous response or an asynchronous response based on the data volume of the access request of the service requester, the response time requirement, and the corresponding response performance index, the method includes:

Specifically, when the interaction mode of the standardized interfaces of the service requester and the service provider is a message interaction mode, it needs to be determined that the message interaction mode is an "online request and synchronous response mode" or an "online request and asynchronous response mode".

Usually, according to the data volume of the access request of the service requester, the response time requirement and the corresponding response performance index, it is determined that the corresponding response mode in the message interaction mode is a synchronous response or an asynchronous response. The online request and synchronous response mode is suitable for data sharing service with fast response data acquisition, small data volume and high response performance. The online request and asynchronous response mode is suitable for preparing data sharing services with slow response data, huge data quantity or low response performance.

Fig. 2 is a schematic diagram illustrating interaction between an online request and a synchronous response provided by the present invention, and as shown in fig. 2, a service requester sends a data request to a service provider, and enters a blocking state to wait for a response data from the service provider to return. And after receiving the data request, the service provider performs a series of business processing to obtain response data and replies the response data to the service requester. And the service requester stops the blocking state and continues to operate after receiving the response data.

The online request and synchronous response mode is suitable for data sharing service with fast response data acquisition, small data volume and high response performance. Since the service requester sending the data request needs to block operation, if the speed of acquiring data by the service provider is slow or the amount of data to be responded is large, the service requester is in a blocking state for a long time, which affects its normal operation.

Although the long-time waiting response problem can be controlled by the service response timeout mechanism, the data interaction using the online request and synchronous response mode in the data sharing service with slow data acquisition or huge data return should still be avoided.

Fig. 3 is a schematic diagram of interaction between an online request and an asynchronous response provided by the present invention, and as shown in fig. 3, a service requester invokes a "prepare data method" of a service provider to send a data request. And after receiving the data request, the service provider returns a request success mark, estimated processing time of data preparation, a time interval for suggesting the condition of regularly inquiring data preparation, a data acquisition mode and other messages. The service requester can continue to operate after obtaining the message, and does not need to be in a blocking state for a long time.

The service provider performs a series of business processes to prepare the data after returning the above-mentioned message.

And after the estimated processing time of the data preparation is reached, the service requester sends a request for inquiring the data preparation condition to the 'inquiring data preparation condition' of the service provider according to the suggested time interval for regularly inquiring the data preparation condition. If the data is not prepared, the service provider returns a message that the data is not prepared, and the service requester continues to inquire after the suggested time interval; if the data is ready, the service provider returns a message containing when and where to get the data.

The service requester can decide when to use a given mode (such as FTP or HTTP protocol) to obtain data in a given valid period according to a given time to a given place (such as FTP or HTTP address).

Fig. 4 is a second schematic diagram illustrating the interaction between the online request and the asynchronous response mode provided by the present invention, and as shown in fig. 4, the service requester calls the "prepare data method" of the service provider to send a data request. And after receiving the data request, the service provider returns a request success mark.

The service requester can continue to operate after obtaining the message, and does not need to be in a blocking state for a long time.

The service provider notifies the service requester after the data is ready.

And after receiving the notification that the data is ready, the service requester initiates a data acquisition request to the service provider and further acquires the data.

The online request and asynchronous response mode is suitable for preparing a data sharing service which responds to data slowly, has huge data quantity or has low response performance. By using the online request and asynchronous response mode, the service requester can normally run other functions without long-time waiting and blocking. For data interaction with large data volume, the service provider can specify a special file transfer protocol, such as FTP or HTTP protocol for transmission.

The data request response interface has the working mode as follows: the time required for the service provider to provide the response file after receiving the request data file should not exceed 24 hours. The method is mainly suitable for the situation that a service requester has specific data screening requirements and a service provider provides data according to the screening requirements.

Optionally, if it is determined that the interaction manner of the plurality of customized standardized interfaces is a publish-subscribe manner, acquiring response data corresponding to a registration request based on the registration request of a service requester, including:

the service provider determining a readiness state of the response data;

Specifically, in the case that the interaction mode of the standardized interfaces of the service requester and the service provider is a publish-subscribe mode, the service requester may periodically obtain subscription data.

Fig. 5 is an interaction diagram of a publish-subscribe mode provided by the present invention, and as shown in fig. 5, a service requester registers a request data request and notification service to a "registration method" of a service provider. And after receiving the registration message of the service requester, the service provider returns a request success mark. The service requester can continue to operate without being in a blocking state for a long time after obtaining the message.

The service provider performs a series of business processes to prepare the data after returning the above message.

When the data preparation is finished, the service provider calls a notification service registered by the service requester to inform the service requester of the data preparation condition and the related acquisition information. The service requester replies with a success or failure flag. The service requester can decide when to use a given mode (such as FTP or HTTP protocol) to obtain data within a given validity period according to a given time to a given place (such as FTP or HTTP address). Thereby enabling triggered interaction of data.

The publish-subscribe mode is suitable for data sharing services for triggerably transmitting various data volumes. Because the publish-subscribe mode is an asynchronous response mode in nature, the service requester does not need to wait and block for a long time and can normally run other functions.

Optionally, if the access request is used for data query, sending the access request to the data processing module based on any one of KPI data query, data model structure query, and DQI data query, including:

determining a data query result corresponding to KPI data query based on dimension attributes and query indexes in KPI data query and Web Service/SOAP technology;

or, based on the data structure name, the dimension index and the Web Service/SOAP technology in the data model structure query, determining the data query result corresponding to the data model structure query;

Specifically, the invention realizes the data query requirement of the external system based on the Web Service/SOAP technology, and simultaneously supports the following relevant technical standards for ensuring the normalization and the interconnection interoperability of the Web Service/SOAP technology:

the whole interface message is based on Extensible Markup Language (XML) Language, and conforms to XML V1.0 (2 nd Edition or highher) specification and XML Schema (1 st Edition or highher) specification.

The interface implementation conforms to the SOAP protocol V1.1 version specification, the high-version SOAP protocol ensures the compatibility with the 1.1 version, and the interface Description uses Web Services Description Language (WSDL).

The interface implementation supports at least the transmission of SOAP over the HTTP V1.0 protocol.

The interface description supports binding of WSDL on the SOAP V1.1 protocol.

And UTF-8 coding mode is adopted in the application layer.

When data query adopts KPI data query, based on metadata, dimension attribute and query index in KPI query as query condition, querying data meeting condition and returning. The method specifically comprises the following steps: and inquiring indexes according to the dimension attributes, inquiring the filtered data according to the index value range, and combining and inquiring the same-dimension multi-index.

And when the data query adopts a data model structure query, querying data meeting the conditions and returning the data based on the metadata, the data structure name in the data model structure query and the dimension index as query conditions. The method specifically comprises the following steps: returning data attributes according to the data structure name, inquiring latitude attributes according to the dimensions, inquiring superior dimensions of the data model according to the dimensions, inquiring indexes according to the dimensions, and inquiring the dimensions according to the indexes.

And when the data query adopts DQI data query, querying the data meeting the condition and returning based on the metadata, the dimension attribute and the query index in the DQI data query as query conditions. The method specifically comprises the following steps: inquiring indexes according to the dimension attributes, inquiring filtering data according to the index value range, combining and inquiring the same-dimension multi-indexes, and inquiring DQI according to the dimension and the inquiry indexes.

In addition, the query result can be displayed through a visual interface by the visual management module, so that a service requester can obtain a more visual query result conveniently.

Through the different data query modes, the corresponding data query service can be flexibly provided for the upper-layer application, and the corresponding query result can be displayed through the visual interface.

Optionally, the data processing module is configured to select a target server cluster for processing the access request of the service requester based on a load balancing algorithm and the determined cluster performance weights of different server clusters in the server clusters, and return corresponding response data from the target server cluster, and the method includes:

based on a load balancing algorithm and the cluster performance weight of each pending server cluster, selecting a target service cluster to process an access request of the service requester;

and sending corresponding response data returned by the target server cluster to the service requester.

Specifically, the data sharing system provided by the present invention further includes a large number of server clusters, which perform calculation for the corresponding processing flows in the data sharing, data acquisition, data query, and the like. And each server may have different configurations, hardware differences and corresponding performances, and a plurality of servers with similar performances form a server cluster, so that the processing capacity of each server is effectively utilized, and the benefit of the server cluster is maximized. Likewise, since the servers with different performances are divided into different server clusters, the different server clusters also correspond to different data processing capacities, i.e. the performances of the different server clusters are different.

And sequencing all the server clusters according to the data processing capacity, the load capacity and the like corresponding to each cluster, and determining the cluster performance weight corresponding to each server cluster according to the sequencing result.

When a high concurrent access request (data sharing request) exists, classifying the data sharing request according to the service type, and screening out a matched pending server cluster in all the service clusters according to the data volume and the response performance requirement. The screened cluster of pending servers herein may include one or more. And under the condition that the undetermined server cluster comprises a plurality of undetermined server clusters, selecting a target service cluster to process the data sharing request by adopting a load balancing algorithm and combining the cluster performance weight of each undetermined server cluster. Specifically, it may be that the higher the cluster performance weight of the pending server cluster is, the higher the probability of being selected as the target service cluster is. But it is also necessary to determine whether the load of the proxy server cluster with a higher cluster performance weight meets the requirement for the data amount in the data sharing request.

Optionally, the system further includes a configuration management module, configured to perform customized configuration on all displayable units included in the visualization management module; wherein the all displayable units include service maps, subscription management, service management, audit management, and shared log management.

Specifically, the data sharing system provided by the invention further comprises a configuration management module, which can be used for customizing the displayable units of the service map, subscription management, service management, audit management, shared log management and the like included in the visualization management module.

The service map is an entrance of a service requester in an external system, provides a visual service navigation function for a third party user of the data sharing system, provides service inquiry according to the type, name and system of the service, and visually displays detailed information of various services.

The types of the services specifically include: FTP-like services and WS-like services.

1) FTP type service:

name of service, data set of service, providing system of service, data scheduling period, path for data storage, etc.

2) Web Services (WS) like services

Name of the service, data set of the service, provisioning system of the service, parameter description of the service, service return data format, and the like.

And subscription management, namely providing management of subscription of the shared service for a service requester of an external system, wherein the service requester of the external system can search the required data shared service through a service map and subscribe the published shared service.

And the service management is used for managing the data sharing service, and the data sharing service supports the third-party application system and the data sharing system. The method mainly comprises the steps of service registration, service change and service logout.

And the service registration is used for issuing the service and providing the service for the service requester to use. The service searching is that the service requester searches the corresponding service in the service registration information, and calls the service according to the searched service binding information.

And the service change provides a changed function for the published service, and modifies the published service content through the service change.

And (4) service logout, namely logging out the unused data service from the system through a logout function, and after the service logout is approved.

And audit management, which provides audit management capability in the whole life cycle of the data sharing service and mainly comprises service registration and approval, service subscription and approval and service cancellation and approval.

And service registration and approval, namely providing an approval function for the newly registered service, and inquiring the approved service in a service map for a third-party system to use.

The service subscription is approved, the subscription service needs to be approved, and the service can be called after approval.

And (4) service cancellation approval, wherein for the cancelled service, the service can be truly cancelled after no user exists in the service.

And sharing log management, wherein the log management records important events in the service operation process and needs to be preserved for a long time and backed up periodically. The log mainly comprises the following contents:

the registration information of the service includes: service registrant information, service identification, service address, service description, service name, registration time and the like;

the service access information includes: service requestor identity information, service identification, service address, service name, data range, etc.

In addition, in the data sharing system provided by the invention, the core hardware equipment needs to adopt redundancy designs such as a distributed system architecture or a thermal system backup, and the like, so that the risk of a single-point fault to the system is avoided.

The method supports the storage facing the application and the configuration and the control of computing resources, realizes the effective isolation between the applications and eliminates the mutual influence of the applications; the system has the capability of automatic application switching, and after the data sharing is abnormal, the system can capture abnormal information and automatically switch to the standby node. And 7-24-hour uninterrupted data sharing and interface service capacity, and a back-up function is provided for data lost in the sharing process.

The automatic exception handling capability is that when a service requester (data consumer) has an exception in the interface service call, the data warehouse management module can capture the exception information and terminate the call, so that the stability and normal work of the data sharing service and the interface are maintained, and other service requesters are not affected. The captured abnormal information can be submitted to manual analysis, and the root cause of the abnormal information is found and solved.

The data sharing system provided by the invention also supports different sharing modes, and specifically comprises the following steps:

mode A: the shared data is completely provided by the data sharing system, and the service requester directly obtains the data required by the subset from the data sharing system;

b mode: each application developer (service request party) self-defines indexes meeting the service requirements based on metadata provided by the data sharing system, and acquires data through a sharing interface;

and (C) mode: each application developer (service request party) customizes the data structure based on the metadata provided by the data sharing system, and obtains and uses the corresponding result data.

Through the different modes, a more flexible data sharing mode is provided for different developers or third-party applications, and the compatibility is strong.

The data sharing system provided by the invention uniformly manages the mode of accessing the data sharing system by an external system through different standardized interfaces, adopts the server cluster, selects the server cluster with matched performance for the access requests of different service requesters with different service types, and improves the expansion performance of the data sharing service of the data sharing system. On the premise of ensuring data safety and reliability, the data sharing is realized more flexibly.

Fig. 6 is a schematic flow chart of a data sharing method provided by the present invention, and as shown in fig. 6, the method includes:

601, determining a shared service interface corresponding to each upper layer application according to the requirement of the upper layer application;

step 602, sending an access request of a service requester based on an interactive mode of a plurality of user-defined standardized interfaces and the shared service interface corresponding to the upper application; the interaction modes of the multiple user-defined standardized interfaces comprise a message interaction mode, a publish-subscribe mode and a database access mode;

603, selecting a target server cluster for processing the access request of the service requester based on a load balancing algorithm and the determined cluster performance weights of different server clusters in the server clusters, and returning corresponding response data by the target server cluster;

and step 604, providing a visual interface for the service requester to display the response data based on the response data obtained by the data processing module.

Specifically, the data sharing method provided by the invention can be oriented to different scenes, and provides differentiated data sharing. Aiming at various applications provided by different manufacturers, data support can be provided for upper-layer applications mainly through three types of differentiated sharing modes. The three types of differentiated sharing modes mainly include: real-time KPI calculation class, real-time detail inquiry class and analysis class non-real-time statistics. The real-time KPI calculation class mainly provides data support for upper-layer application through a real-time signaling KPI, data quality early warning and on-time statistics; the real-time detail query class mainly provides data support for upper-layer application through number query, user record query and complaint fault query; the analysis-type non-real-time statistics is mainly realized by acquiring and presenting indexes, acquiring detailed data and providing data support for upper-layer applications by using the computing/storage resource of the data sharing system.

And the service request or the upper application requirement for providing data support through real-time KPI calculation is mainly characterized in that: the requirement on the real-time performance of data is high, the shared data volume is small, the interaction process is generally unidirectional, and the data acquisition frequency is real-time.

The method provides a service request or an upper application requirement supported by data through a real-time detail query class, and has the following characteristics: the requirement on the real-time performance of data is low, the shared data volume is small, the interaction process is usually bidirectional, and the data acquisition frequency is on demand.

The service request or upper application requirement supported by data provided by analyzing class non-real-time statistics mainly has the following characteristics: the requirement on the real-time performance of data is low, the size of the shared data can be both, the interaction process is usually bidirectional, and the data acquisition frequency is timing.

The three types of differentiated sharing modes are determined according to the requirements of upper-layer applications through corresponding sharing service interface modules in the data sharing system, and the specific requirements of the upper-layer applications can meet any one of the characteristics of the temporal KPI calculation type, the real-time detail query type, the analysis type non-real-time statistics service request providing data support or the requirements of the upper-layer applications; a corresponding shared service interface is determined.

The data sharing method and the data sharing system aim to guarantee the requirements of universality, usability, consistency, stability and the like of data sharing services and interfaces. By means of the standardized interface and the interface protocol, any external application needing to acquire data can be ensured, and the data can be conveniently acquired through the data sharing system only by following the standard requirement of the data sharing interface.

Wherein, standardized interface and interface protocol include: message queue, web Service, FTP, database, socket, etc. Aiming at the interaction mode of the standardized interface, the invention provides three forms: message interaction mode, publish-subscribe mode and database mode.

The message interaction mode is a mode of data interaction between a service requester and a service provider. The message interaction mode is divided into an online request and synchronous response mode and an online request and asynchronous response mode according to the data interaction mode, and the online request and asynchronous response mode respectively correspond to different data volume and response performance indexes. Typically for acquiring shared data once.

The publish-subscribe method is mainly that a service requester can periodically obtain subscription data.

Different database connection protocols can be adopted as long as the database mode aims at different types of databases, and the invention avoids directly exposing the physical model of the database and improves the coupling of the data sharing system through the universal connection protocol.

In the above description, data sharing is provided for an external system mainly through the three standardized interface interaction modes, before that, it needs to be determined which target server cluster is specifically adopted to process the access request, and when selecting a target server cluster, a cluster performance weight of the server cluster, a service type corresponding to the access request, and a load capacity of each server cluster need to be considered, and based on a load balancing algorithm, a final target server cluster is determined. And after the final response data of the target server cluster is obtained, the response data is displayed to the service request party through a visual interface, so that the service request party can more intuitively know the distribution condition of the shared data.

The data sharing method provided by the invention uniformly manages the mode of accessing the data sharing system by an external system through different standardized interfaces, adopts the server cluster, selects the server cluster with matched performance for the access requests of different service requesters with different service types, and improves the expansion performance of the data sharing service of the data sharing system. On the premise of ensuring data safety and reliability, the data sharing is realized more flexibly.

determining that the access request is for data sharing or data querying;

and if the access request is used for data query, sending the access request to the data processing module based on any one of KPI data query, data model structure query and DQI data query.

and if the interaction mode of the plurality of user-defined standardized interfaces is determined to be a database mode, determining an access interface of an external system to the database based on an ODBC or JDBC access mode.

Optionally, if it is determined that the interaction manners of the multiple customized standardized interfaces are message interaction manners, after determining that a corresponding response manner in the message interaction manners is a synchronous response or an asynchronous response based on the data size of the access request of the service requester, the response time requirement, and the corresponding response performance index, the method includes:

the service provider determining a readiness state of the response data;

Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.

Claims

1. A data sharing system, characterized in that the data sharing system comprises: the system comprises a sharing interface module, a data warehouse management module, a data processing module and a visualization management module;

2. The data sharing system according to claim 1, wherein the shared interface module is configured to determine the shared service interface corresponding to each upper layer application according to requirements of the upper layer application, and includes at least one of:

3. The data sharing system according to claim 1, wherein the data warehouse management module is configured to send an access request of a service requester based on interaction modes of multiple customized standardized interfaces and the shared service interface corresponding to the upper application, and includes:

determining that the access request is for data sharing or data querying;

and if the access request is used for data query, the access request is sent to the data processing module based on any one of key performance indicator KPI data query, data model structure query and design quality indicator DQI data query.

4. The data sharing system according to claim 3, wherein the sending the access request to the data processing module based on any one or a combination of more than one of a message interaction manner, a publish-subscribe manner, and a database manner includes:

and if the interactive mode of the plurality of user-defined standardized interfaces is determined to be a database mode, determining an access interface of an external system to the database based on an open database connection ODBC or Java database connection JDBC access mode.

5. The data sharing system of claim 4, wherein if it is determined that the interaction manners of the plurality of customized standardized interfaces are message interaction manners, after determining that the corresponding response manner in the message interaction manners is a synchronous response or an asynchronous response based on the data amount of the access request of the service requester, the response time requirement, and the corresponding response performance index, the method comprises:

6. The data sharing system according to claim 4, wherein if it is determined that the interaction manner of the plurality of customized standardized interfaces is a publish-subscribe manner, acquiring response data corresponding to a registration request based on the registration request of a service requester comprises:

the service provider determining a readiness state of the response data;

7. The data sharing system according to claim 3, wherein if the access request is for data query, the sending the access request to the data processing module based on any one of KPI data query, data model structure query and DQI data query includes:

8. The data sharing system according to claim 1, wherein the data processing module is configured to select a target server cluster for processing the access request of the service requester based on a load balancing algorithm and the determined cluster performance weights of different server clusters in the server clusters, and return corresponding response data from the target server cluster, and includes:

9. The data sharing system according to claim 1, wherein the system further comprises a configuration management module for custom configuring all displayable units included in the visualization management module; wherein the all displayable units include service maps, subscription management, service management, audit management, and shared log management.

10. A method for sharing data, comprising: