CN113094385B - Data sharing fusion platform and method based on software defined open tool set - Google Patents
Data sharing fusion platform and method based on software defined open tool set Download PDFInfo
- Publication number
- CN113094385B CN113094385B CN202110261104.8A CN202110261104A CN113094385B CN 113094385 B CN113094385 B CN 113094385B CN 202110261104 A CN202110261104 A CN 202110261104A CN 113094385 B CN113094385 B CN 113094385B
- Authority
- CN
- China
- Prior art keywords
- data
- platform
- service
- interface
- information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000004927 fusion Effects 0.000 title claims abstract description 99
- 238000000034 method Methods 0.000 title claims abstract description 37
- 238000012545 processing Methods 0.000 claims abstract description 46
- 238000012544 monitoring process Methods 0.000 claims abstract description 41
- 238000013480 data collection Methods 0.000 claims abstract description 11
- 238000002156 mixing Methods 0.000 claims abstract description 11
- 238000007726 management method Methods 0.000 claims description 112
- 238000003860 storage Methods 0.000 claims description 26
- 238000004458 analytical method Methods 0.000 claims description 23
- 238000011156 evaluation Methods 0.000 claims description 18
- 238000013475 authorization Methods 0.000 claims description 15
- 238000004590 computer program Methods 0.000 claims description 14
- 230000008569 process Effects 0.000 claims description 14
- 238000004422 calculation algorithm Methods 0.000 claims description 13
- 238000000605 extraction Methods 0.000 claims description 13
- 230000002776 aggregation Effects 0.000 claims description 12
- 238000004220 aggregation Methods 0.000 claims description 12
- 238000007405 data analysis Methods 0.000 claims description 11
- 238000013524 data verification Methods 0.000 claims description 11
- 238000013507 mapping Methods 0.000 claims description 10
- 238000011157 data evaluation Methods 0.000 claims description 9
- 238000004364 calculation method Methods 0.000 claims description 7
- 238000013523 data management Methods 0.000 claims description 6
- 230000003993 interaction Effects 0.000 claims description 5
- 238000007500 overflow downdraw method Methods 0.000 claims description 5
- 230000002159 abnormal effect Effects 0.000 claims description 4
- 238000013441 quality evaluation Methods 0.000 claims description 4
- 238000012795 verification Methods 0.000 claims description 4
- 230000000007 visual effect Effects 0.000 claims description 4
- 238000007418 data mining Methods 0.000 claims description 3
- 238000003745 diagnosis Methods 0.000 claims description 3
- 238000012423 maintenance Methods 0.000 claims description 3
- 238000013024 troubleshooting Methods 0.000 claims description 3
- 238000010586 diagram Methods 0.000 description 32
- 230000006870 function Effects 0.000 description 11
- 238000005516 engineering process Methods 0.000 description 9
- 230000010354 integration Effects 0.000 description 8
- 238000011161 development Methods 0.000 description 7
- 230000005856 abnormality Effects 0.000 description 4
- 230000002452 interceptive effect Effects 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 239000006185 dispersion Substances 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012552 review Methods 0.000 description 2
- 244000035744 Hura crepitans Species 0.000 description 1
- 230000004075 alteration Effects 0.000 description 1
- 230000000712 assembly Effects 0.000 description 1
- 238000000429 assembly Methods 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000003032 molecular docking Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 238000013468 resource allocation Methods 0.000 description 1
- 230000003238 somatosensory effect Effects 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 230000007704 transition Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/242—Query formulation
- G06F16/2433—Query languages
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3409—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment for performance assessment
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/248—Presentation of query results
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Computational Linguistics (AREA)
- Mathematical Physics (AREA)
- Computer Hardware Design (AREA)
- Quality & Reliability (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a data sharing fusion platform and method based on a software defined open tool set. The scheme comprises a data sharing and blending platform and a data sharing and blending tool; the data sharing and sharing platform comprises a shared open data collection interface, a data computing interface, a component service interface, a data use interface, a data warehouse, an operating environment, an extensible container engine, a metadata management module, a resource catalog management module, a data quality management module and a platform service management and operation monitoring module; the data sharing fusion tool comprises a data processing tool and a data sharing tool. According to the scheme, a data sharing fusion tool is established in a software definition mode, and the data sharing fusion platform is combined to monitor, fuse and analyze scenes of shared fusion data, so that the problems of low efficiency and high cost of information island data are solved.
Description
Technical Field
The invention relates to the technical field of data processing, in particular to a data sharing fusion platform and method based on a software defined open tool set.
Background
The integration and integration of the information resources can form wide data sharing, so that the flow of the data resources and the basic data is accelerated, business innovation among departments is further promoted, and the method has important significance for meeting the public government requirement and cross-department cooperation. In recent years, the related art of fusion and integration of information resources has been rapidly developing. For example, information highways, access to U.S. projects, etc., have driven continued progress in information resource fusion and integration technologies.
Prior to the technology of the patent, the existing technology mainly aims at gathering data resources, and the development and utilization level is extremely low, so that information islands are easy to appear. The information island multisource data analysis sharing using process is complex, and different service systems need different shared data supports. In order to meet the requirements of different business analysis uses, the data in the current sharing platform are basically stored in various data sources in a scattered way, and the different interfaces of the respective storage systems cause complex multi-source data analysis environments, so that the data use efficiency of the sharing platform is low. Meanwhile, a unified data view is lacking in upper-layer application development, so that the use cost is high. Therefore, in the process of constructing a business system, information islands appear, so that multiple copies of data are caused, and high cost required by business cannot be supported.
Disclosure of Invention
In view of the above problems, the invention provides a data sharing fusion platform and a method based on a software defined open tool set, which establish a data sharing fusion tool in a software defined manner, and combine the data sharing fusion platform to monitor, fuse and analyze the shared fusion data, so as to solve the problems of low efficiency and high cost of information island data.
According to a first aspect of an embodiment of the present invention, a data sharing fusion platform based on a software defined open tool set is provided. The platform comprises a data sharing and blending platform and a data sharing and blending tool;
The data sharing and sharing platform comprises a shared open data collection interface, a data computing interface, a component service interface, a data use interface, a data warehouse, an operating environment, an extensible container engine, a metadata management module, a resource catalog management module, a data quality management module and a platform service management and operation monitoring module;
The data sharing fusion tool comprises a data processing tool and a data sharing tool;
the shared open data aggregation interface is used for receiving external multi-source information of the data sharing fusion platform;
the data computing interface is used for feeding the data obtained by computing back to the outside of the data sharing fusion platform;
The component service interface is used for carrying out data interaction between the data sharing fusion platform and the data sharing fusion tool;
And the data use interface expands application of the data after data processing outside the data sharing fusion platform.
In one or more embodiments, preferably, the metadata management module specifically includes:
The metadata storage sub-module is used for managing all interface metadata in the data source interface, the message middleware and the front-end display, and storing the interface metadata in a classified manner according to technical metadata and business metadata;
The metadata query sub-module is used for querying and retrieving the metadata basic information in the metadata storage sub-module and returning query content information, wherein the query content information comprises a database table, a dimension table, indexes, a process and object information participating in input and output;
the metadata checking sub-module is used for checking the integrity and rationality of the metadata information, wherein the checking content comprises middleware sql analysis success rate, table level relation integrity rate and field relation integrity rate;
and the metadata verification sub-module is used for verifying the validity of the parameters by using JSR303 according to the resource directory management module.
In one or more embodiments, preferably, the resource catalog management module specifically includes:
The information resource catalogs are used for inputting department catalogs, basic catalogs and main catalogs, and maintaining and managing the department catalogs, the basic catalogs and the main catalogs, wherein the main catalogs comprise population, legal persons and electronic license;
The government information resources comprise a front section code, a message middleware and a back section code, wherein the front section code is used for representing classification of the government information resources, the back section code is used for representing the sequence of the government information resources, and the message middleware is used for storing message contents;
The directory inquiry sub-module is used for inquiring and checking the corresponding information resources according to the directory of the government information resources, wherein inquiring the government information resources needs to obtain inquiry resource authorization;
And the catalog authority sub-module is used for authorizing different user levels by a manager, wherein the authorization content comprises resource checking authorization, resource inquiring authorization and resource modifying authorization.
In one or more embodiments, preferably, the data quality management module specifically includes:
The data quality management sub-module is used for carrying out quality rule management, quality rule execution, data quality monitoring, quality problem management and quality evaluation report generation;
the data evaluation management sub-module is used for performing data use evaluation, data relationship evaluation, timeliness evaluation, redundant data evaluation and importance evaluation;
and the data auditing management sub-module is used for performing auditing rule management, auditing task scheduling and auditing result analysis.
In one or more embodiments, preferably, the platform service management and operation monitoring module specifically includes:
The platform service management sub-module is used for registering services, discovering services, checking started services and checking service states;
the cluster management sub-module is used for notifying a technical responsible person of troubleshooting problems when the technical responsible person cannot normally use the cluster management sub-module in the process of calling the interface and monitoring the clusters;
the monitoring sub-module is used for providing monitoring information of the exchange tasks in a graphical interface form, wherein the monitoring information comprises task scheduling and real-time flow;
the monitoring alarm sub-module is used for carrying out state diagnosis and display on abnormal, alarm and fault conditions;
The log management sub-module is used for inquiring log contents according to the operation result, the operation type, the operation time and the operation user;
And the performance monitoring sub-module is used for measuring performance data at 5-minute intervals, and carrying out historical data query and real-time data query according to the performance data.
In one or more embodiments, preferably, the data sharing tool specifically includes:
The operation management component is used for collecting and classifying the operation conditions of the data sharing and blending platform and sending the operation condition data to management staff according to a preset operation and maintenance monitoring specification;
the storage management component comprises a distributed file storage and a distributed memory database;
the search management component is used for providing a search entry and performing full-text search, database statistics and record viewing according to the keywords;
The data interface component comprises an authentication interface, a data collection interface component, a data calculation interface component, a data use interface component and a platform service interface, wherein the structures of the authentication interface, the data collection interface component, the data calculation interface component, the data use interface component and the platform service interface all comprise request URL and a request method;
the open source platform assembly library comprises a batch acquisition assembly, a distributed file system, a batch processing engine, a memory type batch processing engine, a primary key query and search assembly, a full text search assembly, an acquisition tool, a distributed message assembly, a storage query assembly, an SQL batch processing engine, a resource scheduling engine, an open source computing frame and a stream processing engine.
In one or more embodiments, preferably, the data processing tool specifically includes:
The data exchange component is used for managing the data exchange nodes and the data exchange rules;
The data verification component is used for verifying the type, the length, whether the type, the length, the accuracy, the range and the format of the data;
The data management component is used for organizing all data, metadata and data of the data sharing and sharing platform into a visual interface to be displayed to an administrator, and supporting the administrator to inquire and update resources, catalogs and metadata on line;
the data processing component is used for acquiring service data from the data sharing platform, and the extraction mode is full extraction or incremental extraction according to the service;
and the data analysis component is used for analyzing each data task and carrying out data mining analysis according to an integrated algorithm, wherein the integrated algorithm comprises a text semantic analysis algorithm and an association rule operation algorithm.
According to a second aspect of the embodiment of the invention, a data sharing fusion method based on a software defined open tool set is provided. The method comprises the following steps:
the data sharing and blending platform acquires all multi-source data information;
Carrying out data sharing and aggregation on the multi-source data information to generate shared open data, wherein the shared open data comprises unencrypted data and encrypted data;
Directly accessing the unencrypted data to the data sharing co-fusion platform, and storing the encrypted data into a target blockchain in a preset format by utilizing the blockchain;
Carrying out data processing of a preset format, a preset classification and a preset interface on the multi-source data information, and storing the multi-source data information into a data warehouse;
performing data aggregation, data fusion and authority management on the data sharing and sharing platform, and storing the multi-source data information into metadata, interface data and multi-source sharing data in a preset form;
creating a structured mapping table for the multi-source data information through a data sharing fusion tool and the data sharing fusion platform;
establishing virtual structure data in the data sharing and sharing platform;
And accessing the structured mapping table by executing SQL sentences, and calling and updating the virtual structure data.
According to a third aspect of embodiments of the present invention, there is provided a computer readable storage medium having stored thereon computer program instructions which, when executed by a processor, implement a method according to any of the first aspect of embodiments of the present invention.
According to a fourth aspect of embodiments of the present invention there is provided an electronic device comprising a memory and a processor, the memory being for storing one or more computer program instructions, wherein the one or more computer program instructions are executable by the processor to implement the steps of any of the first aspects of embodiments of the present invention.
The technical scheme provided by the embodiment of the invention can comprise the following beneficial effects:
1) When the embodiment of the invention opens the data access service for the external visitor, the virtual structure table is supported to be created in the aspect of safety so as to shield the real metadata information and the instance information of the local physical data source for the remote access request.
2) The embodiment of the invention provides unified metadata and SQL access interfaces, before accessing the multi-source database, a structured mapping table is created for the multi-source database through SQL grammar provided by a data sharing fusion tool and a platform, and the data query efficiency is improved by accessing the just-created structured mapping table through executing SQL sentences, so that the application development difficulty is reduced.
3) According to the embodiment of the invention, the collection fusion analysis is carried out on the external data through the data sharing fusion platform, so that the closed loop of the multi-source data processing flow in one platform is realized, the data flow is more efficient, the interactive data query is realized based on the platform, and the data use efficiency is improved.
4) According to the embodiment of the invention, the fusion analysis of the multi-source data in one platform is realized through the data sharing fusion platform, multiple data processing shares the same data, the data is prevented from being moved, the problems of data dispersion and repeated storage are solved, the data query speed is improved, the uniform scheduling of resources is realized through platform service management, and the utilization rate of data resources is improved.
Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and claims thereof as well as the appended drawings.
The technical scheme of the invention is further described in detail through the drawings and the embodiments.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings that are needed in the description of the embodiments will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a block diagram of a data sharing fusion platform based on a software defined open toolset in accordance with one embodiment of the present invention.
FIG. 2 is a schematic illustration of an application of a data sharing fusion platform based on a software defined open toolset according to one embodiment of the present invention.
FIG. 3 is a schematic diagram of a resource data review flow of a resource catalog management module according to one embodiment of the present invention.
FIG. 4 is a schematic diagram of a query flow diagram of a directory query sub-module in one embodiment of the invention.
FIG. 5 is a schematic diagram of a rights setting flow of a directory rights sub-module in one embodiment of the invention.
FIG. 6 is a schematic diagram of an evaluation management flow of a data evaluation management sub-module in one embodiment of the invention.
FIG. 7 is a schematic service management flow diagram of a platform service management sub-module in one embodiment of the invention.
FIG. 8 is a log management flow diagram of a log management sub-module in one embodiment of the invention.
FIG. 9 is a flow chart illustrating the execution of the data verification component in one embodiment of the invention.
FIG. 10 is a schematic diagram of a data parsing flow of a data verification component in an embodiment of the invention.
FIG. 11 is a schematic diagram of a data classification flow of a data verification component in one embodiment of the invention.
FIG. 12 is a flow chart illustrating the execution of a data processing component in one embodiment of the invention.
FIG. 13 is a flow diagram illustrating the execution of a data management component in one embodiment of the invention.
FIG. 14 is a flow chart illustrating the execution of a data analysis component in one embodiment of the invention.
FIG. 15 is a multi-level technology architecture diagram of a data sharing fusion platform based on a software defined open toolset in accordance with one embodiment of the present invention.
FIG. 16 is a flow chart of a method of data sharing fusion based on a software defined open toolset in accordance with one embodiment of the present invention.
Fig. 17 is a block diagram of an electronic device in one embodiment of the invention.
Detailed Description
In some of the flows described in the specification and claims of the present invention and in the foregoing figures, a plurality of operations occurring in a particular order are included, but it should be understood that the operations may be performed out of order or performed in parallel, with the order of operations such as 101, 102, etc., being merely used to distinguish between the various operations, the order of the operations themselves not representing any order of execution. In addition, the flows may include more or fewer operations, and the operations may be performed sequentially or in parallel. It should be noted that, the descriptions of "first" and "second" herein are used to distinguish different messages, devices, modules, etc., and do not represent a sequence, and are not limited to the "first" and the "second" being different types.
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to fall within the scope of the invention.
The integration and integration of the information resources can form wide data sharing, so that the flow of the data resources and the basic data is accelerated, business innovation among departments is further promoted, and the method has important significance for meeting the public government requirement and cross-department cooperation. In recent years, the related art of fusion and integration of information resources has been rapidly developing. For example, information highways, access to U.S. projects, etc., have driven continued progress in information resource fusion and integration technologies.
Prior to the technology of the patent, the existing technology mainly aims at gathering data resources, and the development and utilization level is extremely low, so that information islands are easy to appear. The information island multisource data analysis sharing using process is complex, and different service systems need different shared data supports. In order to meet the requirements of different business analysis uses, the data in the current sharing platform are basically stored in various data sources in a scattered way, and the different interfaces of the respective storage systems cause complex multi-source data analysis environments, so that the data use efficiency of the sharing platform is low. Meanwhile, a unified data view is lacking in upper-layer application development, so that the use cost is high. Therefore, in the process of constructing a business system, information islands appear, so that multiple copies of data are caused, and high cost required by business cannot be supported.
The embodiment of the invention provides a data sharing fusion platform and a data sharing fusion method based on a software defined open tool set. According to the scheme, a data sharing fusion tool is established in a software definition mode, and the data sharing fusion platform is combined to monitor, fuse and analyze scenes of shared fusion data, so that the problems of low efficiency and high cost of information island data are solved.
According to a first aspect of an embodiment of the present invention, a data sharing fusion platform based on a software defined open tool set is provided. FIG. 1 is a block diagram of a data sharing fusion platform based on a software defined open toolset in accordance with one embodiment of the present invention. As shown in fig. 1, the platform includes a data sharing common fusion platform 101 and a data sharing fusion tool 102;
The data sharing and sharing platform 101 comprises a shared open data collection interface 103, a data computing interface 104, a component service interface 105, a data use interface 106, a data warehouse 107, an operating environment 108, an extensible container engine 109, a metadata management module 110, a resource catalog management module 111, a data quality management module 112, a platform service management and operation monitoring module 113;
the data sharing fusion tool includes a data processing tool 114 and a data sharing tool 115;
the shared open data aggregation interface 103 is used for receiving external multi-source information of the data sharing fusion platform;
the data computing interface 104 is configured to feed the data obtained by the computation back to the outside of the data sharing fusion platform;
The component service interface 105 is configured to perform data interaction between the data sharing fusion platform and the data sharing fusion tool;
The data use interface 106 expands application of the data after data processing outside the data sharing fusion platform.
In the embodiment of the invention, the data sharing and blending platform and the data sharing and blending tool realize service registration and discovery between each other through Eureka. The data sharing fusion platform is realized by a micro-service architecture. Eureka is a representative state transition based service that is the underlying component of service registration and discovery. The Eureka client registers own service information on the server, maintains own information consistency, facilitates other services to find themselves, acquires other service information depending on the Eureka server, and completes service call. Specifically, the registration means that the service provider registers its own metadata with the Eureka server side for service discovery. The discovery indicates that the Eureka client obtains other registered service information through the Eureka server, so that the required service is found to initiate remote call.
Therefore, if the data sharing fusion platform needs to call a component in the data sharing fusion tool, firstly, starting the component service, registering own metadata with the Eureka server for service discovery, and when the application is started, registering own service information with the Eureka server by the Eureka client, and caching the service information of the Eureka server to the local. The Eureka client periodically interacts with the Eureka server to update service leases and service information by sending heartbeats to the Eureka server to maintain and update the validity of the service instance metadata in the registry. During a certain period of time, the Eureka server does not receive heartbeat information of the Eureka client, defaults to offline, and service instance information is deleted from the registry. The service instance metadata is actively logged off of the Eureka server at service shutdown, and the service provider instance data is deleted from the Eureka server's registry.
FIG. 2 is a schematic illustration of an application of a data sharing fusion platform based on a software defined open toolset according to one embodiment of the present invention. As shown in fig. 2, the data sharing fusion platform realizes metadata management, resource catalog management, data quality management, platform service management, platform operation monitoring, data aggregation interface, data use interface, data calculation interface and component service interface functions. The data aggregation interface realizes data sharing and aggregation, and shared open data is divided into two types. 1) The unencrypted data is directly imported into the platform by the platform sharable open data attribution interface, 2) the encrypted data is stored into the blockchain by using the blockchain client by taking the intelligent contract as a support, and the data sharing among the data open authority alliance members in the intelligent contract is performed. The data computing interface mainly realizes a docking channel of an external data and data sharing fusion platform, and realizes high-performance computing processing of data. The component service interface mainly realizes that the registration and discovery service of the component is a channel between the data sharing fusion platform and the data sharing fusion tool. The data use interface mainly realizes data sharing fusion platform data application, including interactive application, search application, real-time report form, real-time decision analysis, enterprise credit management, institution financial risk and the like.
In one or more embodiments, preferably, the metadata management module 110 specifically includes:
The metadata storage sub-module is used for managing all interface metadata in the data source interface, the message middleware and the front-end display, and storing the interface metadata in a classified manner according to technical metadata and business metadata;
The metadata query sub-module is used for querying and retrieving the metadata basic information in the metadata storage sub-module and returning query content information, wherein the query content information comprises a database table, a dimension table, indexes, a process and object information participating in input and output;
the metadata checking sub-module is used for checking the integrity and rationality of the metadata information, wherein the checking content comprises middleware sql analysis success rate, table level relation integrity rate and field relation integrity rate;
Specifically, the structured query language (Structured Query Language) is abbreviated as SQL, which is a database query and programming language used for accessing data and querying, updating and managing a relational database system.
And the metadata verification sub-module is used for verifying the validity of the parameters by using JSR303 according to the resource directory management module.
Specifically, JSR is an abbreviation of Java Specification Requests, which represents a Java specification proposal. A formal request form of standardized technical specifications.
In the embodiment of the invention, the data of all the data sources are collected into the storage system of the data sharing fusion platform. All data are stored according to the resource allocation classification. The data under the resource can be consulted through specific conditions when the query is carried out. Firstly, target management is needed to realize the management of department directories, basic directories, main body directories and government information resource directories of all departments. Inside the catalog are all government information resources. Including population, legal, electronic license underlying directory management. The government information resource code structure is composed of a front section code, a message middleware and a back Duan Mazu. The front section code consists of class, item, fine and fine code and is used as a classification code of government information resource; message middleware is the actual transmission content; the back-end code is a sequence code of the government information resource. The directory query group is used for realizing multidimensional directory query and checking authorized information resources under the directory according to the result ordering. The directory permission management is used for authorizing management personnel for different user levels, and meeting permission requirements of different users on directory browsing and searching.
FIG. 3 is a schematic diagram of a resource data review flow of a resource catalog management module according to one embodiment of the present invention. As shown in fig. 3, in one or more embodiments, preferably, the resource catalog management module 111 specifically includes:
The information resource catalogs are used for inputting department catalogs, basic catalogs and main catalogs, and maintaining and managing the department catalogs, the basic catalogs and the main catalogs, wherein the main catalogs comprise population, legal persons and electronic license;
The government information resources comprise a front section code, a message middleware and a back section code, wherein the front section code is used for representing classification of the government information resources, the back section code is used for representing the sequence of the government information resources, and the message middleware is used for storing message contents;
The directory inquiry sub-module is used for inquiring and checking the corresponding information resources according to the directory of the government information resources, wherein inquiring the government information resources needs to obtain inquiry resource authorization;
And the catalog authority sub-module is used for authorizing different user levels by a manager, wherein the authorization content comprises resource checking authorization, resource inquiring authorization and resource modifying authorization.
FIG. 4 is a schematic diagram of a query flow diagram of a directory query sub-module in one embodiment of the invention. As shown in fig. 4, by selecting different data classifications and secondary classifications of data, further searching keywords, and realizing multi-dimensional directory inquiry through an inquiry list.
FIG. 5 is a schematic diagram of a rights setting flow of a directory rights sub-module in one embodiment of the invention. As shown in fig. 5, the user performs by role selection, matches different types of rights groups, and further performs management of data rights according to the rights group types. Finally, the manager is authorized for different user levels, and the permission requirements of different users for browsing and searching the catalogue are met.
In one or more embodiments, preferably, the data quality management module 112 specifically includes:
The data quality management sub-module is used for carrying out quality rule management, quality rule execution, data quality monitoring, quality problem management and quality evaluation report generation;
the data evaluation management sub-module is used for performing data use evaluation, data relationship evaluation, timeliness evaluation, redundant data evaluation and importance evaluation;
and the data auditing management sub-module is used for performing auditing rule management, auditing task scheduling and auditing result analysis.
In the embodiment of the invention, the implementation comprises supporting quality rule management, quality rule execution, data quality monitoring, quality problem management and quality evaluation reporting. Data assessment management is used to implement data usage assessment, data relationship assessment, timeliness assessment, redundant data assessment, and importance assessment. The data auditing management is used for realizing auditing rule management, auditing task scheduling and auditing result analysis.
FIG. 6 is a schematic diagram of an evaluation management flow of a data evaluation management sub-module in one embodiment of the invention. Through data validity judgment, legal data classification and illegal data marking are carried out, further, combination condition selection of data attributes is carried out, customized data is formed, and further, data use evaluation, data relation evaluation, timeliness evaluation, redundant data evaluation and importance evaluation are realized.
In one or more embodiments, preferably, the platform service management and operation monitoring module 113 specifically includes:
The platform service management sub-module is used for registering services, discovering services, checking started services and checking service states;
the cluster management sub-module is used for notifying a technical responsible person of troubleshooting problems when the technical responsible person cannot normally use the cluster management sub-module in the process of calling the interface and monitoring the clusters;
the monitoring sub-module is used for providing monitoring information of the exchange tasks in a graphical interface form, wherein the monitoring information comprises task scheduling and real-time flow;
the monitoring alarm sub-module is used for carrying out state diagnosis and display on abnormal, alarm and fault conditions;
The log management sub-module is used for inquiring log contents according to the operation result, the operation type, the operation time and the operation user;
And the performance monitoring sub-module is used for measuring performance data at 5-minute intervals, and carrying out historical data query and real-time data query according to the performance data.
In an embodiment of the present invention, platform services manage functions for managing various services, including registering/discovering services, viewing all services of the system, viewing services that have been initiated by the current system, viewing the service status of a specified service. The platform operation monitor is used for cluster management, collection monitoring, monitoring alarm, log management and performance monitoring.
Specifically, the cluster management aims at a series of problems that an interface cannot be normally used or cluster pressure is too high to normally provide service due to a certain reason in the process of calling the interface and monitoring the system clusters, and related responsible persons must be notified and processed. And realizing node management and cluster monitoring management.
Specifically, the collection monitoring is used for providing a platform manager with a graphical interface form to monitor the exchange task, and mainly comprises task scheduling monitoring and real-time flow monitoring. The method and the system realize the display of the information and the state of the front-end processor in a graphic mode, wherein the information and the state comprise the running state, the system performance and the resource occupation condition of each front-end processor, the log and the abnormal condition of the system, and the running condition of the switching service deployed by each front-end switching subsystem.
FIG. 7 is a schematic service management flow diagram of a platform service management sub-module in one embodiment of the invention. As shown in fig. 7, by checking and applying for resources by creator and manager, checking and executing personnel, creating and publishing, confirming and publishing, judging service publishing state and sandbox test result by tester, and finally giving final result by observer, further supporting the functions of managing various services, including registering/discovering service, checking all services of system; checking the started service of the current system; the service status of the specified service is checked.
FIG. 8 is a log management flow diagram of a log management sub-module in one embodiment of the invention. As shown in fig. 8, more detailed log information is obtained by querying information and selecting corresponding logs, in this process, a fixed query result is returned according to query conditions, and log detail viewing is provided; finally, the user operation log can be checked, and the user operation log comprises records and displays of contents such as user login, authority change, IP, operation type, operation content, operation time and the like; specifically, the system log viewing is supported, and the system log comprises records and displays of contents such as system startup, system shutdown, user increase and decrease, user login, authority change, performance abnormality, function abnormality, hardware abnormality, network abnormality, dangerous operation and the like.
In one or more embodiments, the data sharing tool 115 preferably specifically includes:
The operation management component is used for collecting and classifying the operation conditions of the data sharing and blending platform and sending the operation condition data to management staff according to a preset operation and maintenance monitoring specification;
the storage management component comprises a distributed file storage and a distributed memory database;
the search management component is used for providing a search entry and performing full-text search, database statistics and record viewing according to the keywords;
The data interface component comprises an authentication interface, a data collection interface component, a data calculation interface component, a data use interface component and a platform service interface, wherein the structures of the authentication interface, the data collection interface component, the data calculation interface component, the data use interface component and the platform service interface all comprise request URL and a request method;
the open source platform assembly library comprises a batch acquisition assembly, a distributed file system, a batch processing engine, a memory type batch processing engine, a primary key query and search assembly, a full text search assembly, an acquisition tool, a distributed message assembly, a storage query assembly, an SQL batch processing engine, a resource scheduling engine, an open source computing frame and a stream processing engine.
FIG. 9 is a flow chart illustrating the execution of the data verification component in one embodiment of the invention. As shown in fig. 9, the data verification content supports data verification on information such as type, length, null, precision, range, format, and the like.
FIG. 10 is a schematic diagram of a data parsing flow of a data verification component in an embodiment of the invention. As shown in fig. 10, the data parsing provides a data type checking function before warehousing after extracting data such as relational data, non-relational data, data files, FTP message middleware and the like in the distributed and heterogeneous data sources to the temporary intermediate layer, and a high-performance data parsing function for mass data.
FIG. 11 is a schematic diagram of a data classification flow of a data verification component in one embodiment of the invention. As shown in fig. 11, when data collection occurs, data collected by each service department is transmitted to a data center pre-library system to be verified, the pre-library verifies whether the item information is consistent, if so, the comparison is successful, otherwise, the comparison error is returned, and the accurate information collected by each service department is returned to the caller. When synchronizing data to the business department, the data check is used for comparing the consistency of the imported data and the business department data, if the data are consistent, the data do not need to be imported repeatedly, and if the data are inconsistent, the inconsistent information is covered by the latest data. And the data check comparison tool is used for displaying the result of checking comparison to a user in a report form convenient for reading and analyzing.
FIG. 12 is a flow diagram illustrating the execution of a data management component in one embodiment of the invention. In fig. 12, the data management module mainly organizes all the data, metadata, information of the data and the like under the exchange platform into a visual interface for display to an administrator, and the administrator can complete the management work of inquiring/updating the resources, the catalogs and the metadata on line. When data exchange is carried out, the data task can acquire related resource configuration to carry out operations such as detection and verification on the data according to the resource ID of the data.
FIG. 13 is a flow chart illustrating the execution of a data processing component in one embodiment of the invention. As shown in fig. 13, the data processing component supports the retrieval of business data from a data sharing platform. The data sources are a service system and a file system, the extraction mode supports full extraction or increment extraction according to specific services, and the extraction time and frequency are formulated according to specific services, so that the parameters are supported to be configured.
FIG. 14 is a flow chart illustrating the execution of a data analysis component in one embodiment of the invention.
As shown in fig. 14, in one or more embodiments, the data processing tool 114 preferably specifically includes:
The data exchange component is used for managing the data exchange nodes and the data exchange rules;
The data verification component is used for verifying the type, the length, whether the type, the length, the accuracy, the range and the format of the data;
The data management component is used for organizing all data, metadata and data of the data sharing and sharing platform into a visual interface to be displayed to an administrator, and supporting the administrator to inquire and update resources, catalogs and metadata on line;
the data processing component is used for acquiring service data from the data sharing platform, and the extraction mode is full extraction or incremental extraction according to the service;
and the data analysis component is used for analyzing each data task and carrying out data mining analysis according to an integrated algorithm, wherein the integrated algorithm comprises a text semantic analysis algorithm and an association rule operation algorithm.
FIG. 15 is a multi-level technology architecture diagram of a data sharing fusion platform based on a software defined open toolset in accordance with one embodiment of the present invention. As shown in fig. 15, the data sharing fusion platform based on the software defined open tool set includes a plurality of technology levels, including a data resource layer, a data acquisition layer, a data fusion layer and a data service layer. The method comprises the steps of acquiring data at a data resource layer, performing data interaction service at a data acquisition layer, preprocessing, storing and analyzing technical data at a data fusion layer, forming an application function at a data service layer and coordinating fusion analysis.
According to a second aspect of the embodiment of the invention, a data sharing fusion method based on a software defined open tool set is provided. FIG. 16 is a flow chart of a data sharing fusion method based on a software defined open toolset of the present invention. As shown in fig. 16, the method includes:
s1601, a data sharing and sharing fusion platform acquires all multi-source data information;
S1602, carrying out data sharing and aggregation on the multi-source data information to generate shared open data, wherein the shared open data comprises unencrypted data and encrypted data;
s1603, directly accessing the unencrypted data to the data sharing co-fusion platform, and storing the encrypted data into a target blockchain in a preset format by using the blockchain;
S1604, carrying out data processing of a preset format, a preset classification and a preset interface on the multi-source data information, and storing the multi-source data information into a data warehouse;
s1605, carrying out data aggregation, data fusion and authority management on the data sharing and sharing platform, and storing the multi-source data information into metadata, interface data and multi-source sharing data in a preset form;
S1606, creating a structured mapping table for the multi-source data information through a data sharing fusion tool and the data sharing fusion platform;
s1607, establishing virtual structure data in the data sharing and sharing platform;
s1608, accessing the structured mapping table by executing SQL sentences, and calling and updating the virtual structure data.
In the embodiment of the invention, the data sharing fusion platform solves the problem of low efficiency of multi-source data access analysis in the data sharing process, realizes unified access of one SQL (structured query language) connected with a plurality of data sources, realizes consistent metadata model, consistent authority model, consistent access model and consistent management model, and realizes unified management of multi-source easily purchased data source information. Based on the open platform and the tool set of data sharing, the data access layer accesses the original data interface into the data sharing platform by the platform operation unit. And providing a service system data interface reconstruction service to form a data interface corresponding to the service. The data sharing fusion tool solves the problem of high cost of a programmable data analysis processing environment, the platform provides a series of assembly tools for data processing and a large number of completed algorithm assemblies, the environment corresponding to the data sharing open platform is butted in a service mode, a data user can adopt a modularized development method, the processing and analysis efficiency of data is improved, and the problem of high construction cost of a business data analysis processing environment is solved.
According to a third aspect of embodiments of the present invention, there is provided a computer readable storage medium having stored thereon computer program instructions which, when executed by a processor, implement a method according to any of the first aspect of embodiments of the present invention.
According to a fourth aspect of an embodiment of the present invention, there is provided an electronic device. Fig. 17 is a block diagram of an electronic device in one embodiment of the invention. The electronic device shown in fig. 17 is a general-purpose data sharing fusion apparatus including a general-purpose computer hardware structure including at least a processor 1701 and a memory 1702. The processor 1701 and the memory 1702 are connected by a bus 1703. The memory 1702 is adapted to store instructions or programs executable by the processor 1701. The processor 1701 may be a stand-alone microprocessor or may be a set of one or more microprocessors. Thus, the processor 1701 executes the instructions stored in the memory 1702 to thereby perform the method flow of the embodiment of the present invention as described above to achieve processing of data and control of other devices. The bus 1703 connects the above components together and connects the above components to the display controller 1704 and display devices and input/output (I/O) devices 1705. Input/output (I/O) devices 1705 may be a mouse, keyboard, modem, network interface, touch input device, somatosensory input device, printer, and other devices known in the art. Typically, the input/output devices 1705 are connected to the system through input/output (I/O) controllers 1706.
The technical scheme provided by the embodiment of the invention can comprise the following beneficial effects:
1) When the embodiment of the invention opens the data access service for the external visitor, the virtual structure table is supported to be created in the aspect of safety so as to shield the real metadata information and the instance information of the local physical data source for the remote access request.
2) The embodiment of the invention provides unified metadata and SQL access interfaces, before accessing the multi-source database, a structured mapping table is created for the multi-source database through SQL grammar provided by a data sharing fusion tool and a platform, and the data query efficiency is improved by accessing the just-created structured mapping table through executing SQL sentences, so that the application development difficulty is reduced.
3) According to the embodiment of the invention, the collection fusion analysis is carried out on the external data through the data sharing fusion platform, so that the closed loop of the multi-source data processing flow in one platform is realized, the data flow is more efficient, the interactive data query is realized based on the platform, and the data use efficiency is improved.
4) According to the embodiment of the invention, the fusion analysis of the multi-source data in one platform is realized through the data sharing fusion platform, multiple data processing shares the same data, the data is prevented from being moved, the problems of data dispersion and repeated storage are solved, the data query speed is improved, the uniform scheduling of resources is realized through platform service management, and the utilization rate of data resources is improved.
It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, magnetic disk storage, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
It will be apparent to those skilled in the art that various modifications and variations can be made to the present invention without departing from the spirit or scope of the invention. Thus, it is intended that the present invention also include such modifications and alterations insofar as they come within the scope of the appended claims or the equivalents thereof.
Claims (8)
1. The data sharing fusion platform based on the software defined open tool set is characterized by comprising a data sharing fusion platform and a data sharing fusion tool;
the data sharing and sharing platform comprises a shared open data collection interface, a data computing interface, a component service interface, a data use interface, a data warehouse, an operating environment, an extensible container engine, a metadata management module, a resource catalog management module, a data quality management module and a platform service management and operation monitoring module; the data sharing fusion tool comprises a data processing tool and a data sharing tool; the shared open data aggregation interface is used for receiving external multi-source information of the data sharing fusion platform; the data computing interface is used for feeding the data obtained by computing back to the outside of the data sharing fusion platform; the component service interface is used for carrying out data interaction between the data sharing fusion platform and the data sharing fusion tool; the data use interface expands application of the data after data processing outside the data sharing fusion platform;
The data sharing tool specifically comprises: the operation management component is used for collecting and classifying the operation conditions of the data sharing and blending platform and sending the operation condition data to management staff according to a preset operation and maintenance monitoring specification; the storage management component comprises a distributed file storage and a distributed memory database; the search management component is used for providing a search entry and performing full-text search, database statistics and record viewing according to the keywords; the data interface component comprises an authentication interface, a data collection interface component, a data calculation interface component, a data use interface component and a platform service interface, wherein the structures of the authentication interface, the data collection interface component, the data calculation interface component, the data use interface component and the platform service interface all comprise request URL and a request method; the open source platform assembly library comprises a batch acquisition assembly, a distributed file system, a batch processing engine, a memory type batch processing engine, a main key inquiry and retrieval assembly, a full text retrieval assembly, an acquisition tool, a distributed message assembly, a storage inquiry assembly, an SQL batch processing engine, a resource scheduling engine, an open source computing frame and a stream processing engine;
The data processing tool specifically comprises: the data exchange component is used for managing the data exchange nodes and the data exchange rules; the data verification component is used for verifying the type, the length, whether the type, the length, the accuracy, the range and the format of the data; the data management component is used for organizing all data, metadata and data of the data sharing and sharing platform into a visual interface to be displayed to an administrator, and supporting the administrator to inquire and update resources, catalogs and metadata on line; the data processing component is used for acquiring service data from the data sharing platform, and the extraction mode is full extraction or incremental extraction according to the service; the data analysis component is used for analyzing each data task and carrying out data mining analysis according to an integrated algorithm, wherein the integrated algorithm comprises a text semantic analysis algorithm and an association rule operation algorithm;
The data sharing co-fusion platform and the data sharing fusion tool realize service registration and discovery between each other through Eureka; the data sharing fusion platform is realized by a micro-service architecture; the Eureka client registers own service information on a server, and acquires other service information which is relied on by the Eureka server to finish service call after maintaining own information consistency, and registers metadata representing that a service provider registers the own service information to the Eureka server for service discovery; finding out other service information which indicates that the Eureka client acquires registration through the Eureka server, so as to find out the required service to initiate remote call;
If a component in the data sharing fusion tool is required to be called by the data sharing fusion platform, firstly, starting component service, registering metadata of the component to a Eureka server for service discovery, and when an application is started, registering own service information to the Eureka server by the Eureka client, and caching the service information of the Eureka server to the local; the Eureka client periodically performs heartbeat interaction with the Eureka server to update service lease and service information, and the validity of the service instance metadata in the registry is maintained and updated by sending heartbeats to the Eureka server; the Eureka server does not receive heartbeat information of the Eureka client in a certain period of time, defaults off line, and service instance information is deleted from the registry; the service instance metadata is actively logged off of the Eureka server at service shutdown, and the service provider instance data is deleted from the Eureka server's registry.
2. The data sharing fusion platform based on a software defined open tool set according to claim 1, wherein the metadata management module specifically comprises:
The metadata storage sub-module is used for managing all interface metadata in the data source interface, the message middleware and the front-end display, and storing the interface metadata in a classified manner according to technical metadata and business metadata;
The metadata query sub-module is used for querying and retrieving the metadata basic information in the metadata storage sub-module and returning query content information, wherein the query content information comprises a database table, a dimension table, indexes, a process and object information participating in input and output;
the metadata checking sub-module is used for checking the integrity and rationality of the metadata information, wherein the checking content comprises middleware sql analysis success rate, table level relation integrity rate and field relation integrity rate;
and the metadata verification sub-module is used for verifying the validity of the parameters by using JSR303 according to the resource directory management module.
3. The data sharing fusion platform based on the software defined open tool set according to claim 1, wherein the resource catalog management module specifically comprises:
The information resource catalogs are used for inputting department catalogs, basic catalogs and main catalogs, and maintaining and managing the department catalogs, the basic catalogs and the main catalogs, wherein the main catalogs comprise population, legal persons and electronic license;
The government information resources comprise a front section code, a message middleware and a back section code, wherein the front section code is used for representing classification of the government information resources, the back section code is used for representing the sequence of the government information resources, and the message middleware is used for storing message contents;
The directory inquiry sub-module is used for inquiring and checking the corresponding information resources according to the directory of the government information resources, wherein inquiring the government information resources needs to obtain inquiry resource authorization;
And the catalog authority sub-module is used for authorizing different user levels by a manager, wherein the authorization content comprises resource checking authorization, resource inquiring authorization and resource modifying authorization.
4. The data sharing fusion platform based on the software defined open tool set according to claim 1, wherein the data quality management module specifically comprises:
The data quality management sub-module is used for carrying out quality rule management, quality rule execution, data quality monitoring, quality problem management and quality evaluation report generation;
the data evaluation management sub-module is used for performing data use evaluation, data relationship evaluation, timeliness evaluation, redundant data evaluation and importance evaluation;
and the data auditing management sub-module is used for performing auditing rule management, auditing task scheduling and auditing result analysis.
5. The data sharing fusion platform based on the software defined open tool set according to claim 1, wherein the platform service management and operation monitoring module specifically comprises:
The platform service management sub-module is used for registering services, discovering services, checking started services and checking service states;
the cluster management sub-module is used for notifying a technical responsible person of troubleshooting problems when the technical responsible person cannot normally use the cluster management sub-module in the process of calling the interface and monitoring the clusters;
the monitoring sub-module is used for providing monitoring information of the exchange tasks in a graphical interface form, wherein the monitoring information comprises task scheduling and real-time flow;
the monitoring alarm sub-module is used for carrying out state diagnosis and display on abnormal, alarm and fault conditions;
The log management sub-module is used for inquiring log contents according to the operation result, the operation type, the operation time and the operation user;
And the performance monitoring sub-module is used for measuring performance data at 5-minute intervals, and carrying out historical data query and real-time data query according to the performance data.
6. A data sharing fusion method based on a software defined open tool set, wherein the method is applied to the data sharing fusion platform according to any one of claims 1 to 5; the method comprises the steps of:
the data sharing and blending platform acquires all multi-source data information;
Carrying out data sharing and aggregation on the multi-source data information to generate shared open data, wherein the shared open data comprises unencrypted data and encrypted data;
Directly accessing the unencrypted data to the data sharing co-fusion platform, and storing the encrypted data into a target blockchain in a preset format by utilizing the blockchain;
Carrying out data processing of a preset format, a preset classification and a preset interface on the multi-source data information, and storing the multi-source data information into a data warehouse;
performing data aggregation, data fusion and authority management on the data sharing and sharing platform, and storing the multi-source data information into metadata, interface data and multi-source sharing data in a preset form;
creating a structured mapping table for the multi-source data information through a data sharing fusion tool and the data sharing fusion platform;
establishing virtual structure data in the data sharing and sharing platform;
And accessing the structured mapping table by executing SQL sentences, and calling and updating the virtual structure data.
7. A computer readable storage medium having stored thereon computer program instructions, which when executed by a processor implement the data sharing fusion platform of any of claims 1-5.
8. An electronic device comprising a memory and a processor, wherein the memory is configured to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the data sharing fusion platform of any of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110261104.8A CN113094385B (en) | 2021-03-10 | 2021-03-10 | Data sharing fusion platform and method based on software defined open tool set |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110261104.8A CN113094385B (en) | 2021-03-10 | 2021-03-10 | Data sharing fusion platform and method based on software defined open tool set |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113094385A CN113094385A (en) | 2021-07-09 |
| CN113094385B true CN113094385B (en) | 2024-04-30 |
Family
ID=76666826
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110261104.8A Active CN113094385B (en) | 2021-03-10 | 2021-03-10 | Data sharing fusion platform and method based on software defined open tool set |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113094385B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113656608B (en) * | 2021-08-18 | 2023-10-24 | 中国科学院软件研究所 | Big data system and automatic data processing method for software defined satellite |
| CN113742357A (en) * | 2021-08-25 | 2021-12-03 | 国核电力规划设计研究院有限公司 | Method and system for automatically collecting and associating cross-platform design data |
| CN114780261B (en) * | 2022-04-28 | 2022-11-29 | 四川虹魔方网络科技有限公司 | Basic data micro-service sharing system and method in intelligent equipment operation management |
| CN116720300B (en) * | 2023-08-02 | 2023-10-13 | 福州水字节科技有限公司 | Drainage pipe network model system |
| CN117033487B (en) * | 2023-08-11 | 2024-05-07 | 北京长河数智科技有限责任公司 | System and method for flexibly arranging interfaces based on data sharing |
Citations (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102722667A (en) * | 2012-03-07 | 2012-10-10 | 甘肃省电力公司信息通信公司 | Database security protection system and method based on virtual databases and virtual patches |
| CN103246724A (en) * | 2013-05-08 | 2013-08-14 | 昆明能讯科技有限责任公司 | Open enterprise information portal supporting component-based development and management |
| CN104123133A (en) * | 2014-07-01 | 2014-10-29 | 曾安里 | Dragon VVR-VBF modularization development framework platform tool system |
| CN105554070A (en) * | 2015-12-09 | 2016-05-04 | 北京中科云集科技有限公司 | Method based on police affair big data center service construction |
| CN106777227A (en) * | 2016-12-26 | 2017-05-31 | 河南信安通信技术股份有限公司 | Multidimensional data convergence analysis system and method based on cloud platform |
| CN106855962A (en) * | 2015-12-09 | 2017-06-16 | 星际空间（天津）科技发展有限公司 | A kind of method for building government affairs big data platform |
| CN107491510A (en) * | 2017-08-03 | 2017-12-19 | 国网江苏省电力公司信息通信分公司 | One kind mixing heterogeneous data source unified query system and distributed enquiring method |
| CN109766374A (en) * | 2018-12-26 | 2019-05-17 | 科大国创软件股份有限公司 | A kind of credit joint supervising platform |
| CN111274294A (en) * | 2020-01-09 | 2020-06-12 | 中国科学院计算机网络信息中心 | Universal distributed heterogeneous data integrated logic convergence organization, release and service method and system |
| CN111400749A (en) * | 2020-03-06 | 2020-07-10 | 江苏荣泽信息科技股份有限公司 | Government affair financial data sharing platform based on block chain and implementation method thereof |
| CN111625510A (en) * | 2020-05-25 | 2020-09-04 | 广东电网有限责任公司 | Multi-source data sharing system and method based on cloud mapping |
| CN111930355A (en) * | 2020-08-12 | 2020-11-13 | 创意信息技术股份有限公司 | Web back-end development framework and construction method thereof |
| CN112116488A (en) * | 2020-04-28 | 2020-12-22 | 刘革瑞 | Water conservancy big data comprehensive maintenance system |
| CN112307396A (en) * | 2020-10-21 | 2021-02-02 | 五凌电力有限公司 | Platform architecture based on multi-engine data modeling calculation analysis and processing method thereof |
| CN112306992A (en) * | 2020-11-04 | 2021-02-02 | 内蒙古证联信息技术有限责任公司 | Big data platform based on internet |
| CN112379653A (en) * | 2020-12-01 | 2021-02-19 | 国能信控互联技术有限公司 | Smart power plant management and control system based on micro-service architecture |
| CN112396404A (en) * | 2020-11-27 | 2021-02-23 | 广州光点信息科技有限公司 | Data center system |
-
2021
- 2021-03-10 CN CN202110261104.8A patent/CN113094385B/en active Active
Patent Citations (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102722667A (en) * | 2012-03-07 | 2012-10-10 | 甘肃省电力公司信息通信公司 | Database security protection system and method based on virtual databases and virtual patches |
| CN103246724A (en) * | 2013-05-08 | 2013-08-14 | 昆明能讯科技有限责任公司 | Open enterprise information portal supporting component-based development and management |
| CN104123133A (en) * | 2014-07-01 | 2014-10-29 | 曾安里 | Dragon VVR-VBF modularization development framework platform tool system |
| CN105554070A (en) * | 2015-12-09 | 2016-05-04 | 北京中科云集科技有限公司 | Method based on police affair big data center service construction |
| CN106855962A (en) * | 2015-12-09 | 2017-06-16 | 星际空间（天津）科技发展有限公司 | A kind of method for building government affairs big data platform |
| CN106777227A (en) * | 2016-12-26 | 2017-05-31 | 河南信安通信技术股份有限公司 | Multidimensional data convergence analysis system and method based on cloud platform |
| CN107491510A (en) * | 2017-08-03 | 2017-12-19 | 国网江苏省电力公司信息通信分公司 | One kind mixing heterogeneous data source unified query system and distributed enquiring method |
| CN109766374A (en) * | 2018-12-26 | 2019-05-17 | 科大国创软件股份有限公司 | A kind of credit joint supervising platform |
| CN111274294A (en) * | 2020-01-09 | 2020-06-12 | 中国科学院计算机网络信息中心 | Universal distributed heterogeneous data integrated logic convergence organization, release and service method and system |
| CN111400749A (en) * | 2020-03-06 | 2020-07-10 | 江苏荣泽信息科技股份有限公司 | Government affair financial data sharing platform based on block chain and implementation method thereof |
| CN112116488A (en) * | 2020-04-28 | 2020-12-22 | 刘革瑞 | Water conservancy big data comprehensive maintenance system |
| CN111625510A (en) * | 2020-05-25 | 2020-09-04 | 广东电网有限责任公司 | Multi-source data sharing system and method based on cloud mapping |
| CN111930355A (en) * | 2020-08-12 | 2020-11-13 | 创意信息技术股份有限公司 | Web back-end development framework and construction method thereof |
| CN112307396A (en) * | 2020-10-21 | 2021-02-02 | 五凌电力有限公司 | Platform architecture based on multi-engine data modeling calculation analysis and processing method thereof |
| CN112306992A (en) * | 2020-11-04 | 2021-02-02 | 内蒙古证联信息技术有限责任公司 | Big data platform based on internet |
| CN112396404A (en) * | 2020-11-27 | 2021-02-23 | 广州光点信息科技有限公司 | Data center system |
| CN112379653A (en) * | 2020-12-01 | 2021-02-19 | 国能信控互联技术有限公司 | Smart power plant management and control system based on micro-service architecture |
Non-Patent Citations (2)
| Title |
|---|
| From databases to dataspaces: a new abstraction for information management;Michael Franklin等;ACM SIGMOD Record;20051201;第34卷(第4期);27-33 * |
| 一种面向配电网调度业务管控的混合型大数据平台研究;殷自力等;供用电;20201105;37(11);15-20 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113094385A (en) | 2021-07-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113094385B (en) | Data sharing fusion platform and method based on software defined open tool set | |
| CN110765337B (en) | Service providing method based on internet big data | |
| CN112396404A (en) | Data center system | |
| US8832662B2 (en) | Rules engine for architectural governance | |
| CN109582717B (en) | Database unified platform for electric power big data and reading method thereof | |
| JP4327481B2 (en) | Database system, server, inquiry input method and data update method | |
| US10216782B2 (en) | Processing of updates in a database system using different scenarios | |
| CN108255712A (en) | The test system and test method of data system | |
| CN110458678B (en) | Financial data verification method and system based on hadoop verification | |
| US20190310985A1 (en) | Data arrangement management in a distributed data cluster environment of a shared pool of configurable computing resources | |
| CN114925045B (en) | PaaS platform for big data integration and management | |
| US8904357B2 (en) | Dashboard for architectural governance | |
| US9123006B2 (en) | Techniques for parallel business intelligence evaluation and management | |
| US20190050435A1 (en) | Object data association index system and methods for the construction and applications thereof | |
| US20200250587A1 (en) | Framework for multi-tenant data science experiments at-scale | |
| Varga et al. | Introducing Microsoft SQL Server 2016: Mission-Critical Applications, Deeper Insights, Hyperscale Cloud | |
| CN113535846B (en) | Big data platform and construction method thereof | |
| CN109829615B (en) | Target task multistage monitoring device and method based on proprietary cloud | |
| CN111538720A (en) | Method and system for cleaning basic data in power industry | |
| CN115168474B (en) | Internet of things central station system building method based on big data model | |
| CN111414355A (en) | Offshore wind farm data monitoring and storing system, method and device | |
| US10942924B2 (en) | LOB query performance via automatic inference of locator-less LOB by value semantics | |
| US9305066B2 (en) | System and method for remote data harmonization | |
| US20140136274A1 (en) | Providing multiple level process intelligence and the ability to transition between levels | |
| CN110493326B (en) | Zookeeper-based cluster configuration file management system and method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |