CN115242879B

CN115242879B - Data sharing system and method

Info

Publication number: CN115242879B
Application number: CN202210764896.5A
Authority: CN
Inventors: 赵晓芬; 李林; 张博
Original assignee: Inspur Communication Technology Co Ltd
Current assignee: Inspur Communication Technology Co Ltd
Priority date: 2022-06-29
Filing date: 2022-06-29
Publication date: 2024-04-02
Anticipated expiration: 2042-06-29
Also published as: CN115242879A

Abstract

The invention provides a data sharing system and a method, wherein the system comprises the following steps: the system comprises a sharing interface module, a data warehouse management module, a data processing module and a visual management module; the shared interface module is used for determining a corresponding shared service interface according to the requirements of the upper layer application; the data warehouse management module is used for sending an access request of a service requester based on the interaction modes of various custom standardized interfaces and the shared service interface corresponding to the upper layer application; the data processing module is used for selecting a target server cluster based on a load balancing algorithm and cluster performance weights of different server clusters and returning response data; the visual interface presentation response data is provided to the service requester. The invention manages the mode of accessing the system in a unified way through different standardized interfaces, adopts the server cluster, selects the server cluster with matched performance for the access requests of different service requesters with different service types, and improves the expansion performance of the data sharing service.

Description

Data sharing system and method

Technical Field

The present invention relates to the field of data sharing technologies, and in particular, to a data sharing system and method.

Background

With the rapid development of the internet and informatization technology, particularly under the background of big data, each industry has an own information application platform or big data platform, and the requirements for data summarization and sharing are very high.

In view of the diversity of demands of clients of system applications for data sharing and the frequency of demand changes, the sharing mode and the sharing interface need to be frequently modified, and certain difficulties are brought to system platform developers, especially to operation and maintenance personnel.

How to effectively utilize the data sharing platform and help operation and maintenance personnel to efficiently solve the demands of clients on the high concurrency sharing of data of the platform has very important significance.

Disclosure of Invention

The invention provides a data sharing system and a data sharing method, which are used for solving the problems that in the prior art, clients need frequent modification of sharing modes and sharing interfaces due to the diversity of data sharing requirements and the frequency of requirement change, are inconvenient to operate and manage, and cannot provide proper data sharing services according to corresponding requirements.

The existing information application platform or data sharing platform cannot provide flexible support for the diversity of data sharing demands and the frequency of demand change of clients of different applications, needs to frequently modify sharing modes and sharing interfaces, has certain operation difficulty in actual operation and maintenance, and is easy to make mistakes.

The invention provides a data sharing system, comprising: the system comprises a sharing interface module, a data warehouse management module, a data processing module and a visual management module;

the shared interface module is used for determining a shared service interface corresponding to each upper-layer application according to the requirements of the upper-layer application;

the data warehouse management module is used for sending an access request of a service requester based on the interaction modes of various custom standardized interfaces and the shared service interface corresponding to the upper layer application; the interaction modes of the plurality of custom standardized interfaces comprise a message interaction mode, a publishing and subscribing mode and a database access mode;

the data processing module is used for selecting a target server cluster for processing the access request of the service requester based on a load balancing algorithm and the determined cluster performance weight of different server clusters in the server clusters, and returning corresponding response data by the target server cluster;

the visual management module is used for providing a visual interface for the service requester to display the response data based on the response data obtained by the data processing module.

According to the data sharing system provided by the invention, the sharing interface module is used for determining the sharing service interface corresponding to each upper layer application according to the requirement of the upper layer application, and comprises at least one of the following:

based on the access authority corresponding to the third party application, controlling whether the third party application is accessed to the data sharing system or not, and providing a channel encryption mechanism;

or, determining an available interface of each tenant in the multi-tenant based on the access right and the access amount requirement of the multi-tenant;

or, determining whether to limit access to the third party application based on the access frequency of the third party application;

or adjusting the processing priority of the access requests with different priorities based on the priority order of the access requests in the access queue and the connection state of the access requests.

According to the data sharing system provided by the invention, the sending of the access request to the data processing module based on any one or more of a message interaction mode, a publish-subscribe mode and a database mode comprises:

if the interaction modes of the plurality of custom standardized interfaces are determined to be message interaction modes, determining that the corresponding response modes in the message interaction modes are synchronous response or asynchronous response based on the data volume of the access request of the service requester, the response time requirement and the corresponding response performance index;

If the interaction modes of the plurality of custom standardized interfaces are determined to be the publishing and subscribing modes, acquiring response data corresponding to a registration request based on the registration request of a service requester;

if the interaction mode of the plurality of custom standardized interfaces is a database mode, determining an access interface of an external system to the database based on an ODBC or JDBC access mode.

According to the data sharing system provided by the invention, if it is determined that the interaction modes of the plurality of custom standardized interfaces are message interaction modes, after determining that the corresponding response modes in the message interaction modes are synchronous or asynchronous responses based on the data volume, response time requirement and corresponding response performance indexes of the access request of the service requester, the data sharing system comprises:

if the corresponding response mode in the message interaction mode is synchronous response, the service requester enters a blocking state to wait for the response of the service provider;

and if the corresponding response mode in the message interaction mode is asynchronous response, the service requester determines whether the response data is ready or not through polling or event notification.

According to the data sharing system provided by the invention, if the interaction mode of the plurality of custom standardized interfaces is determined to be a publish-subscribe mode, response data corresponding to a registration request is obtained based on the registration request of a service requester, and the data sharing system comprises:

The service provider determining a readiness of the response data;

and if the service provider determines that the preparation state of the response data is successful, periodically feeding back the response data to the service requester according to the feedback data mode and the feedback period included in the registration request.

According to the data sharing system provided by the present invention, if the access request is used for data query, the access request is sent to the data processing module based on any one of KPI data query, data model structure query and DQI data query, including:

determining a data query result corresponding to the KPI data query based on dimension attributes and query indexes in the KPI data query and Web Service/Simple Object Access Protocol (SOAP) technology;

or determining a data query result corresponding to the data model structure query based on the data structure name, the dimension index and the Web Service/SOAP technology in the data model structure query;

or determining a data query result corresponding to the DQI data query based on the dimension attribute and the query index in the DQI data query and the Web Service/SOAP technology.

According to the data sharing system provided by the invention, the data processing module is used for selecting a target server cluster for processing the access request of the service requester based on a load balancing algorithm and the determined cluster performance weights of different server clusters in the server clusters, and returning corresponding response data by the target server cluster, and comprises the following steps:

determining one or more matched pending server clusters in all the service clusters based on the service type, data volume or response performance requirements corresponding to the access request of the service requester;

selecting a target service cluster to process the access request of the service requester based on a load balancing algorithm and a cluster performance weight of each pending server cluster;

and sending corresponding response data returned by the target server cluster to the service requester.

The data sharing system provided by the invention further comprises a configuration management module, wherein the configuration management module is used for carrying out customized configuration on all displayable units included in the visual management module; wherein all displayable units include service maps, subscription management, service management, audit management, and shared log management.

The invention provides a data sharing method, which comprises the following steps:

determining a shared service interface corresponding to each upper-layer application according to the requirements of the upper-layer application;

based on the interaction modes of various custom standardized interfaces and the shared service interfaces corresponding to the upper-layer applications, sending an access request of a service requester; the interaction modes of the plurality of custom standardized interfaces comprise a message interaction mode, a publishing and subscribing mode and a database access mode;

selecting a target server cluster for processing the access request of the service requester based on a load balancing algorithm and the determined cluster performance weights of different server clusters in the server clusters, and returning corresponding response data by the target server cluster;

and providing a visual interface for the service requester to display the response data based on the response data obtained by the data processing module.

According to the data sharing system and the method, the mode of accessing the data sharing system by an external system is managed uniformly through different standardized interfaces, the server clusters are adopted, the server clusters with matched performance are selected for the access requests of different service requesters with different service types, and the expansion performance of the data sharing service of the data sharing system is improved. On the premise of guaranteeing the safety and reliability of data, the sharing of the data is realized more flexibly.

Drawings

In order to more clearly illustrate the invention or the technical solutions of the prior art, the following description will briefly explain the drawings used in the embodiments or the description of the prior art, and it is obvious that the drawings in the following description are some embodiments of the invention, and other drawings can be obtained according to the drawings without inventive effort for a person skilled in the art.

FIG. 1 is a schematic diagram of a data sharing system according to the present invention;

FIG. 2 is a schematic diagram showing interaction between an online request and a synchronous response mode provided by the invention;

FIG. 3 is a schematic diagram showing interaction between an online request and an asynchronous response mode provided by the present invention;

FIG. 4 is a second exemplary diagram of the interaction of an on-line request and an asynchronous response scheme provided by the present invention;

FIG. 5 is a schematic diagram of a publish-subscribe scheme provided by the present invention;

fig. 6 is a flow chart of a data sharing method provided by the invention.

Detailed Description

For the purpose of making the objects, technical solutions and advantages of the present invention more apparent, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is apparent that the described embodiments are some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.

The data sharing system and method of the present invention are described below in conjunction with fig. 1-6.

FIG. 1 is a schematic diagram of a data sharing system according to the present invention; as shown in fig. 1, the data sharing system 100 includes: a shared interface module 110, a data warehouse management module 120, a data processing module 130, and a visualization management module 140;

and the visual management module provides a visual interface for the service requester to display the response data based on the response data obtained by the data processing module.

Specifically, the data sharing system provided by the invention can provide differentiated data sharing for different scenes. Aiming at a plurality of applications provided by different manufacturers, data support can be provided for upper-layer applications mainly through three types of differential sharing modes. The three types of differentiated sharing modes mainly comprise: real-time KPI calculation class, real-time detail query class and analysis class non-real-time statistics. The real-time KPI calculation is mainly to provide data support for upper-layer application through real-time signaling KPI, data quality early warning and punctual statistics; the real-time detail query class mainly provides data support for upper-layer application through number query, user record query and complaint fault query; the analysis type non-real-time statistics is mainly presented through index acquisition, detail data acquisition, and an application using a data sharing system to calculate/store resources provides data support for an upper application.

And the service request or upper layer application requirement for providing data support through real-time KPI calculation mainly has the following characteristics: the method has the advantages of high requirement on data instantaneity, small shared data volume, unidirectional interaction process and real-time data acquisition frequency.

The service request or upper layer application requirement for providing data support through the real-time detail query class mainly has the following characteristics: the method has the advantages of low requirement on data instantaneity, small shared data volume, bidirectional interaction process and data acquisition frequency as required.

The service request or upper layer application demand for providing data support by analyzing non-real-time statistics is mainly characterized by comprising the following steps: the method has low requirement on data instantaneity, the size of the shared data volume is enough, the interaction process is usually bidirectional, and the data acquisition frequency is timed.

The three types of differentiated sharing modes are determined according to the requirements of upper-layer applications through corresponding sharing service interface modules in the data sharing system, and the specific requirements of the upper-layer applications can be any type of characteristics meeting the requirements of the time KPI calculation type, the real-time detail query type, the analysis type non-real-time statistics service request providing data support or the upper-layer application requirements; a corresponding shared service interface is determined.

Optionally, the shared interface module is configured to determine, according to a requirement of an upper layer application, a shared service interface corresponding to each upper layer application, where the shared service interface includes at least one of the following:

Specifically, through the characteristics of the upper layer application requirements, after the corresponding shared service interfaces are determined, the authority, the access frequency, the access quantity and the like corresponding to the upper layer application requirements are also required to be determined, whether the shared data can be accessed through the determined shared service interfaces or not is determined, and then the shared service interfaces are uniformly managed.

For example, the access authority corresponding to the third party application is obtained, and whether the third application has the authority for accessing the data sharing system is determined, so that the third party application with the access authority is allowed to access the data sharing system, and the third party application without the access authority for accessing the data sharing system is not allowed to access. And different encryption mechanisms are set according to different weights, so that confidentiality of data is flexibly realized.

Or, a plurality of tenants lease part of servers in the data sharing system, and the tenants have different access rights to different servers, and can independently start or stop the service of a certain tenant without affecting each other through dynamically setting the access rights of each tenant. The needed resources can be dynamically allocated and recovered according to the demands of tenants (upper-layer applications), including CPU computing resources, I/O bandwidth, memory, parallelism, space size and time window; the management capability of I/O bandwidth, CPU and memory is particularly important. Dynamic provisioning of resources may also be performed according to the service level agreements (Service Level Agreement, SLAs) of the tenants. The SLA level here may include: different times, different applications, different jobs, different users, etc.

Or determining whether the attack is malicious or not according to the access frequency of the third-party application, and if the attack is malicious, preventing the third-party application from accessing the data sharing system, so as to ensure the performance and the safety of the data warehouse and the server. If the access frequency is greater than a preset threshold, the malicious attack is determined, and the preset threshold can be obtained according to a statistical value in practical application or set according to experience.

Or, by using different access queues of the device, storing access requests with different priorities, and monitoring the connection state of the access requests in each access queue, the data sharing system is ensured to meet the processing delay requirement of the corresponding access requests under high load. And if a certain access request is connected with the overtime, releasing the corresponding server resource.

The access volume requirement is the number of concurrent access requests in a unit time, and can support at least 50 concurrent processes; when the file mode is adopted for interaction, files with larger data volume (such as more than 10M) can be compressed and then transmitted.

Optionally, the data warehouse management module is configured to send an access request of a service requester based on interaction modes of multiple custom standardized interfaces and the shared service interface corresponding to the upper layer application, where the access request includes:

Determining that the access request is for data sharing or data querying;

if the access request is used for data sharing, sending the access request to the data processing module based on any one or a combination of a message interaction mode, a publish-subscribe mode and a database mode;

and if the access request is used for data query, sending the access request to the data processing module based on any one of KPI data query, data model structure query and DQI data query.

Specifically, the data sharing system provided by the invention aims to ensure the requirements of the data sharing service and the interface, such as universality, usability, consistency, stability and the like. Through the custom standardized interface and interface protocol, any external application needing to acquire data is ensured, and the data can be conveniently acquired through the data sharing system only by following the standard requirement of the data sharing interface.

The custom standardized interface and interface protocol include: message queues, web services, file transfer protocol (File Transfer Protocol, FTP), database access interfaces, socket sockets, etc. The message queue may be divided into a master message queue and a plurality of slave message queues, wherein the message queue currently being processed is the master message queue and the priority of the access request is higher. And the master message queue and the slave message queue can be mutually converted. I.e. the master message queue at the current time may be the slave message queue at the next time.

And the corresponding interface protocol satisfies the following conditions:

the whole interface message is based on XML language and accords with the XML V1.0 (2 nd Edition or Higher) specification and the XML Schema (1 st Edition or Higher) specification.

The interface implementation conforms to the simple object access protocol (Simple Object Access Protocol, SOAP) V1.1 version specification, the high version SOAP protocol ensuring compatibility with version 1.1, the interface description using web services description language (Web Services Description Language, WSDL).

The interface implementation supports at least SOAP transport over the hypertext transfer protocol (Hyper Text Transfer Protocol, HTTP) V1.0 protocol.

The interface implementation accords with the WSDL language V1.1 version specification, and the high-version WSDL language ensures the compatibility with the 1.1 version.

The interface description supports binding of WSDL on the SOAP V1.1 protocol.

Aiming at the interaction mode of the custom standardized interface, the invention provides three forms: a message interaction mode, a publish-subscribe mode and a database mode.

The message interaction mode is a mode that a service requester and a service provider interact data. The message interaction mode is divided into an online request and synchronous response mode and an online request and asynchronous response mode according to the data interaction mode, and the two modes correspond to different data amounts and response performance indexes respectively. Typically for a single acquisition of shared data.

The publishing and subscribing mode is mainly that a service requester can periodically acquire subscription data.

The database mode is to different types of databases, and different database connection protocols can be adopted, and the invention avoids directly exposing the physical model of the database and improves the coupling of the data sharing system through the universal connection protocol.

The above description mainly provides data sharing for the external system through the interaction mode of the three standardized interfaces, and of course, the access request data of the service requester can also be collected, stored, analyzed and the like.

Any system internal data to be shared can realize data sharing without affecting the system data architecture, and the loose coupling mode design is adopted for the shared data and the system internal data. The system should have perfect data abstraction and encapsulation capability, and internal data can be encapsulated as application-oriented data sharing services. The shared data is completely separated from the internal data, and the internal data model is not changed due to the change of the shared data model.

In addition, the data warehouse management module in the data sharing system can also provide data query functions for external systems. Specifically, the data warehouse management module in the invention adopts a Web Service/simple object access protocol (Simple Object Access Protocol, SOAP) technology to realize the data query requirement of an external system.

And the metadata, namely the minimum data unit, is determined, so that the processing between different platforms or applications is facilitated to be performed in a unified mode through the unified minimum data unit. The format of the shared data is defined by metadata. The service requesters realizing the upper layer application and the third party application can access the data according to the description information of the data content and the structure under the condition that the data organization structure in the data sharing system is not clear.

And providing the query result of the data through three preset query modes. The three preset inquiry modes are as follows: key performance index (Key Performance Indicator, KPI) data queries, data model structure queries, and design quality index (Design Quality Indicator, DQI) data queries.

Optionally, the sending the access request to the data processing module based on any one or a combination of a message interaction mode, a publish-subscribe mode and a database mode includes:

if the plurality of custom standardized interfaces are determined to be message interaction modes, determining that the corresponding response modes in the message interaction modes are synchronous response or asynchronous response based on the data volume, response time requirements and corresponding response performance indexes of the access request of the service requester;

If the multiple custom standardized interfaces are determined to be in a publishing and subscribing mode, acquiring response data corresponding to a registration request based on the registration request of a service requester;

if the multiple custom standardized interfaces are determined to be in a database mode, determining an access interface of an external system to the database based on an ODBC or JDBC access mode.

Specifically, the interaction modes of the plurality of custom standardized interfaces comprise a message interaction mode, a publish-subscribe mode, a database mode and the like, and correspond to different custom standardized interfaces and different processing flows.

For example, the interaction modes of the multiple custom standardized interfaces are message interaction modes, and the corresponding response modes in the message interaction modes are determined to be synchronous response or asynchronous response based on the data volume of the access request of the service requester, the response time requirement and the corresponding response performance index. For example, the data volume of the access request is small, the response time requirement is short, the response performance index is high, and a synchronous response mode is generally adopted; if the data size of the access request is large and the response performance index is relatively low, an asynchronous response mode is generally adopted.

Or determining that the interaction mode of the plurality of custom standardized interfaces is a publish-subscribe mode, and acquiring response data corresponding to a registration request based on the registration request of a service requester; for example, if the registration request includes feedback data, the service provider confirms whether the feedback data is supported by itself, and if so, returns response data satisfying the feedback data.

If the interaction mode of the plurality of custom standardized interfaces is a database mode, determining an access interface of an external system to the database based on an open database connection (Open Database Connectivity, ODBC) or Java database connection (Java Database Connectivity, JDBC) access mode.

After determining that the interaction mode of the standardized interfaces of the service requester and the service provider is a database mode, a unified access mode can be provided for an external system through a standard database connection protocol ODBC or JDBC, so that the physical model of the database is prevented from being exposed, and attack and tampering of a third party are avoided. Because the change of the physical model of the database can directly affect the upper layer application, the coupling degree of the data sharing system can be improved by adopting the standard database connection protocol ODBC or JDBC provided by the invention.

Optionally, if the plurality of custom standardized interfaces are determined to be message interaction manners, after determining that the corresponding response manner in the message interaction manner is synchronous response or asynchronous response based on the data volume of the access request of the service requester, the response time requirement and the corresponding response performance index, the method includes:

In particular, in the case where the interaction mode of the standardized interfaces of the service requester and the service provider is a message interaction mode, it is necessary to determine that the message interaction mode is an "online request and synchronous response mode" or an "online request and asynchronous response mode".

The corresponding response mode in the message interaction mode is usually determined to be synchronous response or asynchronous response according to the data volume of the access request of the service requester, the response time requirement and the corresponding response performance index. The online request and synchronous response mode is suitable for acquiring the data sharing service with faster response data, smaller data volume and higher response performance. The online request and asynchronous response mode is suitable for preparing data sharing service with slower response data, huge data quantity or lower response performance.

Fig. 2 is a schematic diagram of interaction between an online request and a synchronous response manner, as shown in fig. 2, in which a service request sends a data request to a service provider, and enters a blocking state to wait for response data of the service provider to return. After receiving the data request, the service provider performs a series of service processes to obtain response data, and replies the response data to the service requester. After receiving the response data, the service requester stops the blocking state to continue operation.

The online request and synchronous response mode is suitable for acquiring the data sharing service with faster response data, smaller data volume and higher response performance. Because the service requester sending the data request needs to block operation, if the speed of the service provider obtaining the data is slower or the response data amount is huge, the service requester can be in a blocking state for a long time, and the normal operation of the service requester is affected.

Although the long-waiting response problem can be controlled by a service response timeout mechanism, the use of online request and synchronous response modes for data interaction in data sharing services where data acquisition is slow or the amount of returned data is huge should still be avoided.

FIG. 3 is a schematic diagram showing interaction between an online request and an asynchronous response mode, wherein the service requester invokes a "data preparation method" of a service provider to send a data request, as shown in FIG. 3. After receiving the data request, the service provider returns a request success flag, estimated processing time of data preparation, time interval for suggesting to query the data preparation condition regularly, data acquisition mode and other information. The service requester can continue to operate after obtaining the message without being in a blocking state for a long time.

The service provider performs a series of service processes to prepare the data after returning the message.

After the estimated processing time of the data preparation arrives, the service requester sends a request for inquiring the data preparation condition to the 'inquiring the data preparation condition' of the service provider according to the suggested time interval for periodically inquiring the data preparation condition. If the data is not ready, the service provider returns a message that the data is not ready, and the service requester continues to inquire after the suggested time interval; if the data is ready, the service provider returns a message containing when, where, how to obtain the data.

The service requester can decide by itself when to fetch data in a given way (e.g., FTP or HTTP protocol) within a given validity period, at a given time to a given location (e.g., FTP or HTTP address).

FIG. 4 is a second schematic diagram of interaction between an online request and an asynchronous response mode provided by the present invention, as shown in FIG. 4, a service requester invokes a "prepare data method" of a service provider to send a data request. The service provider returns a request success flag after receiving the data request.

The service requester can continue to operate after obtaining the message without being in a blocking state for a long time.

The service provider notifies the service requester after the data is ready.

After receiving the notification of data preparation, the service requester initiates a data acquisition request to the service provider and further acquires the data.

The online request and asynchronous response mode is suitable for preparing data sharing service with slower response data, huge data quantity or lower response performance. By using the online request and asynchronous response modes, the service requester does not need to wait and block for a long time, and other functions can be normally operated. For large data volume data interactions, the service provider may specify a dedicated file transfer protocol, such as FTP or HTTP, for transfer.

The working mode of the data request response type interface is as follows: the service provider should not provide the response file more than 24 hours after receiving the request data file. The method is mainly suitable for the situation that a service requester has specific data screening requirements, and a service provider provides data according to the screening requirements.

Optionally, if the interaction mode of the plurality of custom standardized interfaces is determined to be a publish-subscribe mode, obtaining response data corresponding to a registration request based on the registration request of a service requester includes:

the service provider determining a readiness of the response data;

In particular, in the case where the interaction manner of the standardized interfaces of the service requester and the service provider is a publish-subscribe manner, the service requester may periodically obtain subscription data.

Fig. 5 is a schematic diagram of interaction of a publish-subscribe mode provided by the present invention, where as shown in fig. 5, a service request registers a request data requirement and a notification service for a "registration method" of a service provider. The service provider returns a request success flag after receiving the registration message of the service requester. The service requester can continue to operate after obtaining the message without being in a blocked state for a long time.

When the data is ready, the service provider invokes a notification service registered by the service requester to inform the service requester of the data preparation and related acquisition information. The service requester replies with a sign of success or failure. The service requester can decide by itself when to fetch data in a given way (e.g., FTP or HTTP protocol) within a given validity period, at a given time to a given location (e.g., FTP or HTTP address). Thereby enabling triggered interaction of the data.

The publishing and subscribing mode is suitable for the data sharing service which can triggerably transmit various data volumes. Because the publish-subscribe mode is an asynchronous response mode in nature, the service requester does not need to wait and block for a long time, and other functions can be normally operated.

Optionally, if the access request is used for data query, sending the access request to the data processing module based on any one of KPI data query, data model structure query and DQI data query, including:

determining a data query result corresponding to the KPI data query based on dimension attributes and query indexes in the KPI data query and a Web Service/SOAP technology;

Specifically, the invention realizes the data query requirement of the external system based on the Web Service/SOAP technology, and simultaneously supports the following related technical standards for ensuring the standardization and the interconnection interoperability of the Web Service/SOAP technology:

the whole interface message is based on an extensible markup language (Extensible Markup Language, XML) language and accords with the XML V1.0 (2 nd Edition or Higher) specification and the XML Schema (1 st Edition or Higher) specification.

The interface implementation conforms to the SOAP protocol V1.1 version specification, the high version SOAP protocol ensures compatibility with version 1.1, and the interface description uses web services description language (Web Services Description Language, WSDL).

The interface implementation supports at least SOAP transport over HTTP V1.0 protocol.

The interface description supports binding of WSDL on the SOAP V1.1 protocol.

And a UTF-8 coding mode is adopted in an application layer.

When the data query adopts KPI data query, based on metadata, dimension attributes and query indexes in the KPI query are used as query conditions, query data meeting the conditions and return. The method specifically comprises the following steps: inquiring the index according to the dimension attribute, inquiring the filtering data according to the index value range, and merging the inquiry with the multi-index of the same dimension.

When the data query adopts the data model structure query, based on the metadata and the data structure name in the data model structure query, the dimension index is used as a query condition, and the data meeting the condition is queried and returned. The method specifically comprises the following steps: and returning data attributes according to the data structure names, inquiring latitude attributes according to dimensions, inquiring superior dimensions of the data model according to dimensions, inquiring indexes according to dimensions, and inquiring dimensions according to indexes.

When the data query adopts the DQI data query, based on the metadata, the dimension attribute and the query index in the DQI data query are used as query conditions, the data conforming to the conditions are queried and returned. The method specifically comprises the following steps: inquiring the index according to the dimension attribute, inquiring the filtering data according to the index value range, inquiring the same-dimension multi-index combination, and inquiring the DQI according to the dimension and the inquiring index.

In addition, the query result can be displayed through the visual interface by the visual management module, so that a service requester can conveniently obtain a more visual query result.

Through the different data query modes, corresponding data query services can be flexibly provided for upper-layer applications, and corresponding query results can be displayed through a visual interface.

Optionally, the data processing module is configured to select a target server cluster for processing the access request of the service requester based on a load balancing algorithm and the determined cluster performance weights of different server clusters in the server clusters, and return corresponding response data by the target server cluster, including:

Specifically, the data sharing system provided by the invention further comprises a large number of server clusters, and the calculation is performed for the corresponding processing flows in the data sharing, the data acquisition, the data query and the like. Each server may have different configurations and different hardware, and the corresponding performance may have different performances, so that a plurality of servers with similar performances form a server cluster, which is beneficial to effectively utilizing the processing capacity of each server and maximizing the benefit of the server cluster. Likewise, because servers of different capabilities are divided into different server clusters, different server clusters also correspond to different data processing capabilities, i.e., the capabilities of different server clusters are different.

And sequencing all the server clusters according to the data processing capacity, the load capacity and the like corresponding to each cluster, and determining the cluster performance weight corresponding to each server cluster according to the sequencing result.

When a high concurrent access request (data sharing request) exists, the data sharing request is classified according to the service type, and then the matched pending server clusters are screened out from all the service clusters by combining the data volume and the response performance requirements. The cluster of pending servers screened out here may include one or more. And under the condition that the number of the pending server clusters is multiple, adopting a load balancing algorithm, and combining the cluster performance weight of each pending server cluster, selecting a target service cluster to process the data sharing request. In particular, the higher the cluster performance weight, which may be a pending server cluster, the greater the likelihood of being selected as a target service cluster. But it is also necessary to combine whether the load of the proxy server cluster with the higher cluster performance weight meets the requirement of the data amount in the data sharing request.

Optionally, the system further comprises a configuration management module, configured to perform customized configuration on all displayable units included in the visualization management module; wherein all displayable units include service maps, subscription management, service management, audit management, and shared log management.

Specifically, the data sharing system provided by the invention further comprises a configuration management module which can be used for carrying out customized configuration on the displayable units such as the service map, subscription management, service management, audit management, sharing log management and the like which are included in the visual management module.

The service map is an entrance of a service requester in an external system, provides a visual service navigation function for a third party user of the data sharing system, provides a service query according to a service type, a service name, a service providing system and the like, and visually displays detailed information of various services.

The types of the services specifically include: FTP-type services and WS-type services.

1) FTP-type service:

information such as the name of the service, the data set of the service, the service providing system, the data scheduling period, the data saving path and the like.

2) Web Services (WS) class of services

The name of the service, the data set of the service, the providing system of the service, the parameter description of the service, the return data format of the service, etc.

Subscription management, which is to provide management of subscription of shared services for service requesters of external systems, and the service requesters of the external systems can search required data shared services through a service map to subscribe published shared services.

Service management provides management of data sharing services, which support both third party application system and data sharing system. Mainly comprises service registration, service change and service cancellation.

And the service registration is used for publishing the service and providing the service for a service requester. The service searching is that a service requester searches corresponding service in the service registration information and calls the service according to the searched service binding information.

Service change, which is to provide change function for released service, and to modify released service content by service change.

Service cancellation, which can cancel the unused data service from the system through the cancellation function, and after the service cancellation approval passes.

And audit management, wherein audit management capability is provided in the whole life cycle of the data sharing service, and the audit management capability mainly comprises service registration approval, service subscription approval and service cancellation approval.

The service registration and approval provides approval function for the newly registered service, and the approved service can be inquired in the service map for the third party system to use.

The service subscription approval is needed, and the service party can call after approval when subscribing the service.

And the service cancellation approval is that the party can truly cancel the cancelled service after the verification that no user exists.

And (3) sharing log management, wherein the log management records important events in the service operation process, and long-term retention and periodic backup are required. The log mainly contains the following:

the registration information of the service includes: service registrant information, service identification, service address, service description, service name, registration time, etc.;

the service access information includes: service requester identity information, service identification, service address, service name, data range, etc.

In addition, in the data sharing system provided by the invention, the core hardware equipment needs to adopt redundant designs such as a persistent distributed system architecture or hot system backup, so that the risk of single-point faults on the system is avoided.

The configuration and control of application-oriented storage and computing resources are supported, effective isolation between applications is realized, and mutual influence is eliminated; the system has the capability of automatically switching the application, and can capture abnormal information and automatically switch to the standby node after the data sharing is abnormal. And the uninterrupted data sharing and interface service capacity is 7 x 24 hours, and a back-filling function is provided for the data lost in the sharing process.

The automatic exception handling capability, when the service requester (data consumer) has an exception on the interface service call, the data warehouse management module can capture the exception information and terminate the call so as to maintain the stability and normal operation of the data sharing service and the interface without affecting other service requesters. The captured abnormal information can be submitted to manual analysis, and the source of the abnormality is found and solved.

The data sharing system provided by the invention also supports different sharing modes, and specifically comprises the following steps:

a mode: the shared data is completely provided by the data sharing system, and the service requester directly acquires the data required by the subset from the data sharing system;

b mode: each application developer (service requester) self-defines indexes meeting service requirements based on metadata provided by the data sharing system and acquires data through a sharing interface;

c mode: each application developer (service requester) performs customization on the data structure based on the metadata provided by the data sharing system, and obtains and uses corresponding result data.

Through the different modes, a more flexible data sharing mode is provided for different developers or third party applications, and the compatibility is strong.

The data sharing system provided by the invention uniformly manages the mode of accessing the data sharing system by the external system through different standardized interfaces, adopts the server clusters, selects the server clusters with matched performance for the access requests of different service requesters with different service types, and improves the expansion performance of the data sharing service of the data sharing system. On the premise of guaranteeing the safety and reliability of data, the sharing of the data is realized more flexibly.

Fig. 6 is a flow chart of a data sharing method provided by the present invention, as shown in fig. 6, the method includes:

step 601, determining a shared service interface corresponding to each upper layer application according to the requirements of the upper layer application;

step 602, sending an access request of a service requester based on interaction modes of various custom standardized interfaces and the shared service interface corresponding to the upper layer application; the interaction modes of the plurality of custom standardized interfaces comprise a message interaction mode, a publishing and subscribing mode and a database access mode;

step 603, selecting a target server cluster for processing the access request of the service requester based on a load balancing algorithm and the determined cluster performance weights of different server clusters, and returning corresponding response data by the target server cluster;

Step 604, providing a visual interface to the service requester to display the response data based on the response data obtained by the data processing module.

Specifically, the data sharing method provided by the invention can provide differentiated data sharing for different scenes. Aiming at a plurality of applications provided by different manufacturers, data support can be provided for upper-layer applications mainly through three types of differential sharing modes. The three types of differentiated sharing modes mainly comprise: real-time KPI calculation class, real-time detail query class and analysis class non-real-time statistics. The real-time KPI calculation is mainly to provide data support for upper-layer application through real-time signaling KPI, data quality early warning and punctual statistics; the real-time detail query class mainly provides data support for upper-layer application through number query, user record query and complaint fault query; the analysis type non-real-time statistics is mainly presented through index acquisition, detail data acquisition, and an application using a data sharing system to calculate/store resources provides data support for an upper application.

In order to ensure the requirements of the data sharing service, the universality, the usability, the consistency, the stability and the like of the interface. Through standardized interfaces and interface protocols, any external application needing to acquire data is guaranteed, and the data can be acquired conveniently through the data sharing system only by following the standard requirements of the data sharing interface.

Wherein the standardized interface and interface protocol comprises: message queues, web Service, FTP, databases, sockets, etc. For the interaction mode of the standardized interface, the invention provides three forms: a message interaction mode, a publish-subscribe mode and a database mode.

The message interaction mode is a mode that a service requester and a service provider interact data. The message interaction mode is divided into an online request and synchronous response mode and an online request and asynchronous response mode according to the data interaction mode, and the two modes correspond to different data amounts and response performance indexes respectively. Typically for one-time acquisition of shared data.

In the foregoing description, the data sharing is provided for the external system mainly through the interaction modes of the three standardized interfaces, before that, it is required to determine which target server cluster is specifically adopted to process the access request, and when selecting the target server cluster, it is required to consider the cluster performance weight of the server cluster, the service type corresponding to the access request, the load of each server cluster, and determine the final target server cluster based on a load balancing algorithm. And after the response data of the final target server cluster is obtained, the response data is displayed to the service requester through a visual interface, so that the service requester can more intuitively know the distribution condition of the shared data.

The data sharing method provided by the invention uniformly manages the mode of accessing the data sharing system by the external system through different standardized interfaces, adopts the server clusters, selects the server clusters with matched performance for the access requests of different service requesters with different service types, and improves the expansion performance of the data sharing service of the data sharing system. On the premise of guaranteeing the safety and reliability of data, the sharing of the data is realized more flexibly.

Optionally, the shared interface module is configured to determine, according to a requirement of an upper application, a shared service interface corresponding to each upper application, where the shared service interface includes at least one of the following:

determining that the access request is for data sharing or data querying;

Optionally, if the interaction mode of the plurality of custom standardized interfaces is determined to be a message interaction mode, after determining that the corresponding response mode in the message interaction mode is a synchronous response or an asynchronous response based on the data volume, the response time requirement and the corresponding response performance index of the access request of the service requester, the method includes:

The service provider determining a readiness of the response data;

Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present invention, and are not limiting; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data sharing system, the data sharing system comprising: the system comprises a sharing interface module, a data warehouse management module, a data processing module and a visual management module;

the visual management module provides a visual interface for the service requester to display the response data based on the response data obtained by the data processing module;

the shared interface module is configured to determine, according to requirements of upper-layer applications, a shared service interface corresponding to each of the upper-layer applications, where the shared service interface includes at least one of the following:

2. The data sharing system according to claim 1, wherein the data warehouse management module is configured to send an access request of a service requester based on interaction modes of multiple custom standardized interfaces and the shared service interface corresponding to the upper layer application, and the data warehouse management module includes:

determining that the access request is for data sharing or data querying;

and if the access request is used for data query, sending the access request to the data processing module based on any one of key performance index KPI data query, data model structure query and design quality index DQI data query.

3. The data sharing system according to claim 2, wherein the sending the access request to the data processing module based on any one or more of a combination of a message interaction mode, a publish-subscribe mode, and a database mode includes:

if the interaction mode of the plurality of custom standardized interfaces is a database mode, determining an access interface of an external system to the database based on an open database connection ODBC or Java database connection JDBC access mode.

4. The data sharing system according to claim 3, wherein, if the interaction mode of the plurality of custom standardized interfaces is determined to be a message interaction mode, after determining that the corresponding response mode in the message interaction mode is a synchronous response or an asynchronous response based on the data volume, the response time requirement and the corresponding response performance index of the access request of the service requester, the method comprises:

5. The data sharing system according to claim 3, wherein if the interaction mode of the plurality of custom standardized interfaces is determined to be a publish-subscribe mode, obtaining response data corresponding to a registration request based on the registration request of a service requester includes:

the service provider determining a readiness of the response data;

6. The data sharing system of claim 2, wherein if the access request is for a data query, sending the access request to the data processing module based on any one of a KPI data query, a data model structure query, and a DQI data query comprises:

7. The data sharing system of claim 1, wherein the data processing module is configured to select a target server cluster for processing the access request of the service requester based on a load balancing algorithm and the determined cluster performance weights of different server clusters in the server clusters, and return corresponding response data from the target server cluster, and comprises:

8. The data sharing system of claim 1, further comprising a configuration management module for customized configuration of all displayable units included in the visualization management module; wherein all displayable units include service maps, subscription management, service management, audit management, and shared log management.

9. A data sharing method, comprising:

providing a visual interface to the service requester to display the response data based on the response data;

according to the requirements of upper-layer applications, determining a shared service interface corresponding to each upper-layer application, wherein the shared service interface comprises at least one of the following components:

based on the access authority corresponding to the third party application, controlling whether the third party application is accessed to a data sharing system or not, and providing a channel encryption mechanism;